Measuring and Communicating the Impact of your Coalition and Programs. Brian Bumbarger Coalition Leadership Institute June 2010

Transcription

1 Measuring and Communicating the Impact of your Coalition and Programs Brian Bumbarger Coalition Leadership Institute June 2010

2 If you don t know where you re going, any road will get you there! The Cheshire Cat, Alice in Wonderland

3 It is not enough to be busy. So are the ants. The question is: What are we busy about? Henry David Thoreau

4 Don t mistake movement for progress. They might feel the same, but just shaking your butt won t get you to Disney World Brian Bumbarger

5 Create community-level outcomes. Investigate potential programs, policies and practices. Select programs, policies and practices. Create programlevel outcomes. Plan to evaluate. Create a written Community Action Plan.

6 Evaluate (v.): to determine the significance, worth, or condition of usually by careful appraisal and study Assess (v.) : to determine the importance, size, or value of

7 Pre-test Prevention Program Post-test We can compare the difference between pre-test and post-test But What if something else caused the change? If the post-test isn t better, does that mean the program didn t work? What would have happened in the absence of the program? Are these kids representative of all kids?

8 Pre-test Prevention Program Post-test

9 Randomization R a n d o m S e l e c t i o n Pre-test Pre-test Prevention Program Prevention Program Post-test Post-test F o l l o w u p D a t a

10 Create community-level outcomes. Investigate potential programs, policies and practices. Select programs, policies and practices. Create programlevel outcomes. Plan to evaluate. Create a written Community Action Plan.

11 Gives clear direction toward achieving community vision Provides built-in evaluation measures and accountability Required by many grant makers

12

13 Decrease in problem behaviors Increase in protective factors Decrease in risk factors Changes in participant knowledge, attitudes, skills or behavior Program implementation fidelity (10 to 15 years) (3 to 10 years) (1 to 4 years) (6 months to 2 years)

14

15

16 The Community Board will learn how to develop outcome-focused prevention plans.

17 Participants will be able to: 1. Define outcome-focused planning. 2. Write desired outcomes: Behavior outcomes Risk-factor outcomes Protective-factor outcomes 3. Prepare for community evaluation.

18 Measuring changes Measuring progress Knowing what actions will take place

19 1. Change in what 2. As measured by 3. The baseline or starting point for comparison 4. By how much, by when

20 1. Priority risk-factor outcome to change 2. Indicator used to measure the outcome 3. Baseline data 4. How much change, by when Community Plan Implementation Training 1-<#>

21 Identify targets for prevention-planning efforts. Evaluate the impact of prevention efforts over time. Engage Key Leaders and community members in supporting prevention. Community Plan Implementation Training 1-<#>

22 Know the data to be collected. Know who is responsible for data collection. Know how outcomes will be reported. Determine what resources will be needed for future assessments.

23 Community Plan Implementation Training 1-<#>

24 Participants will learn how to develop participant and implementation outcomes in preparation for implementing and evaluating their selected programs, policies and practices.

25 Participants will be able to: 1. Develop participant outcomes. 2. Develop implementation outcomes. 3. Identify the elements of implementation.

26 Participant outcomes measure the changes a program produces. Implementation outcomes measure the process by which a program produces desired changes.

27 Knowledge Attitudes Skills Behavior.

28 Who the program will be delivered by When the program will be delivered, including how often and how long Where the program will be delivered How the program will be delivered Number of people to be affected by the program Who your target audience will be.

29 Community Plan Implementation Training 1-<#>

30 Participants will be able to: 1. Identify the uses of evaluation. 2. Explain basic evaluation concepts. 3. Assess the need for an expert evaluator.

31 Evaluation helps you: monitor implementation fidelity monitor progress toward outcomes identify problems with program design or selection demonstrate achievements determine cost-effectiveness.

32 What do you want to evaluate? What is the purpose of your evaluation?

33 Community Board members Key Leaders Program implementers and site administrators Media Local interest groups Other community members Sources of funding

34 Monitor implementation fidelity. Identify and correct implementation problems before they result in program failure.

35 Records Activity logs Observation Questionnaires

36 Measure the program s impact on participant knowledge, skills, attitudes or behavior. Monitor progress toward the community s vision.

37 Observation Interviews Questionnaires Records

38 An expert evaluator can help: select appropriate evaluation design develop evaluation measures analyze statistical information interpret results.

39 The program requires significant technical assistance Evaluation tools will need to be developed Data collection will be expensive and complex Complex data analysis will be required The Board lacks the expertise to complete the evaluation.

40 Local universities Research and consulting firms

41 The nature of the evaluation Who will be doing the evaluation Resources needed for the evaluation.

42 Community Plan Implementation Training 3-42

43 To help ensure that programs, policies and practices in the Community Action Plan are implemented with fidelity. Community Plan Implementation Training 3-43

44 Participants will be able to: 1. Explain the elements of successful, high-fidelity implementation. 2. Develop implementation plans. 3. Identify methods of monitoring implementation for fidelity. Community Plan Implementation Training 3-44

45 High-fidelity implementation means that: the target audience is reached all components of the program are delivered according to the original design the program is delivered at the intended dosage (duration, frequency). Community Plan Implementation Training 3-45

46 Studies have found that: the effectiveness of any program is significantly increased when the program is implemented with fidelity some programs are only effective when implemented with a high degree of fidelity. (Blueprints, 2001) Community Plan Implementation Training 3-46

47 Support of site administrators Qualified implementers who support the program Planning meetings and trainings A detailed implementation plan (who, what, where, when and how) Ongoing monitoring and technical assistance Community Plan Implementation Training 3-47

48 Select implementers who: have the credentials prescribed by the program support the program approach are able to reach/communicate with target audience. Community Plan Implementation Training 3-48

49 Involve implementers in the planning process. Provide implementers with clear information about the program. Train implementers to deliver the program. Consider a written participation agreement. Community Plan Implementation Training 3-49

50 To educate stakeholders and build support for the program To negotiate implementation logistics To develop detailed implementation plans To obtain signed participation agreements Community Plan Implementation Training 3-50

51 Implementers should leave trainings with: a clear understanding of the program goals, rationale, desired outcomes, components and methods confidence that they have the skills and expertise to deliver the program effectively. Community Plan Implementation Training 3-51

52 Include: target audience and expected number of participants plans for recruiting participants who the implementers are schedule for implementing all components. Community Plan Implementation Training 3-52

53 Community Plan Implementation Training 1-<#> 3-53

54 To ensure that the program is implemented with fidelity to the original design. To identify and correct implementation problems. To fulfill requirements of funders. To provide lessons learned for future implementation efforts. To identify and celebrate early successes. Community Plan Implementation Training 3-54

55 The target audience is not participating in great enough numbers. The program components are not being delivered at the intended dosage. Program delivery methods are not being used. Community Plan Implementation Training 3-55

56 Monitor: planning meetings before implementation training of implementers program delivery. Community Plan Implementation Training 3-56

57 Who attended the trainings? What was covered at the trainings? Did implementers leave trainings with the skills, confidence and motivation to deliver the program effectively? Community Plan Implementation Training 3-57

58 How were participants recruited? How many members of the target audience participated? Was the program delivered at the correct dosage? Did implementers use methods prescribed in the program design? Did implementers communicate effectively with the target audience? Community Plan Implementation Training 3-58

59 Community Plan Implementation Training 1-<#> 4-5

60 1. Engage stakeholders. 2. Define what will be evaluated. 3. Select an evaluation design. 4. Decide how you will collect and measure data. 5. Gather, analyze and interpret data. 6. Report your findings and use the results. Community Plan Implementation Training 4-7

61 Use the instrument designed for the program. If no instrument is available, seek help from an expert evaluator. Community Plan Implementation Training 4-15

62 Consider: commitment to the project and to the evaluation objectivity ability to collect complete and accurate data. Community Plan Implementation Training 4-18

63 Determine: who will oversee the evaluation a central storage place for data from each program evaluation deadlines decision-making processes. Community Plan Implementation Training 4-19

64 Choose an appropriate format. Write for your most important audience. Discuss results in terms of the overall program goals. Be honest. Community Plan Implementation Training 4-20

65 Identify individuals with: skills in designing evaluation instruments skills in designing and conducting evaluation studies skills in data analysis and presentation the ability to objectively assess outcome results and recommend possible solutions the ability to credibly communicate results to key stakeholders. Community Plan Implementation Training 4-24

66 Celebrate success. Write a report detailing the results. Fulfill accountability requirements. Identify causes of unmet expectations. Revise and update the Community Action Plan. Community Plan Implementation Training 4-25

67 Weak connection between the program and outcomes Unrealistic outcomes Failure to implement a program with fidelity Participant-related reasons External factors Measurement problems Decreases in sample size (attrition) Community Plan Implementation Training 4-26

68 Celebrate success. Identify changing priorities. Select new tested, effective prevention strategies to address changing priorities. Assist resource allocation decisions. Evaluate policies and practices. Community Plan Implementation Training 4-30

69 If you want to build a ship, don't drum up the men to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea. -- Antoine de Saint-Exupéry

70 Thank You! Brian K. Bumbarger Evidence-based Prevention and Intervention Support Center Prevention Research Center, Penn State University 206 Towers Bldg. University Park, PA (814)

71 Outcome-Based Planning Logic Model Vision (10 15 years) Reduction in problem behavior (3-10 years) Increase in Protective Factors (1-4 years) Decrease in priority risk factors (1-4 years) Programs (6 mos. - 2 years) What kind of future do you envision for your community s children? What changes in ATOD use, delinquency, violence, teen pregnancy, dropout will be necessary to achieve the vision? What changes in protective factors are needed to reduce problem behaviors? What changes in risk factors are needed to reduce problem behaviors? How will your programs & services change risk & protective factors? Vision Statement Problem Behaviors Protective Factor Prevalence Risk Factor Prevalence Program Outcomes