DEMYSTIFYING EVALUATION

Transcription

1 DEMYSTIFYING EVALUATION Make It Plain, Make It Useful, Make It Simple, and Make it Happen! Kristian L. Gordon, MPH, CHES December 8, 2011

2 LEARNING OBJECTIVES Describe the different types of evaluation in nontechnical terms. (Make It Plain) Make the case for program evaluation as an essential element of program sustainability (Make It Useful) Examine the evaluation framework and its application using real world examples. (Make It Simple) Review the South Carolina Online Reporting and Evaluation System (SCORES) and its association with the Obesity State Plan Evaluation. (Make It Happen)

3 MAKE IT PLAIN Describing the different types of evaluation in non-technical terms

4 WHAT IS EVALUATION? Program evaluation is the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming. Patton, M.Q. (1997). Utiliza tion-focused evaluation: The new century text (3 rd ed.). Thousand Oaks, CA: Sage.

5 SYSTEMATIC PROCESS Engage Stakeholders Ensure use and share lessons learned Justify Conclusions Standards Utility Feasibility Propriety Accuracy Describe the Program Focus the evaluation design Gather credible evidence

6 MAKING IT PLAIN What is evaluation? Systematic process Purposeful Grounded in the realities of real world practice Investigative Discovering Revealing Enhancer Information

7 TYPES OF EVALUATION Formative conducted for the purpose of refining or improving a program and is often conducted by people closely involved with the program. Performance Management focuses on activities, direct products and services delivered, and the results of those products or services. Process/Monitoring checking the program s implementation and appropriate use of resources for program improvement, accountability, and the development of effective implementation models.

8 TYPES OF EVALUATION CONTINUED Outcome Evaluation assesses changes in knowledge, attitude, practices as a result of an implemented change. Impact Evaluation assesses what happens as a result of the implemented change such as behavior change. Summative (Outcome/Impact) used to determine the merit, worth, or value in a way that leads to making a final evaluative judgment.

9 EVALUATION JARGON Evaluation Models & Approach Goal-Based Approach Goal-Free Approach Utilization Focused Approach Four-Level Model Evaluation Designs Non-experimental Pre-experimental (survey administered after the completion of a training) Quasi-experimental (Nonrandom assignment) True experimental (Random Assignment) Return on Investment (ROI)

10 QUESTIONS?

11 MAKE IT USEFUL Identifying the role of evaluation in sustainability

12 WHAT IS SUSTAINABILITY? The existence of structures and processes which allow a program to leverage resources to most effectively implement evidence-based polices and activities over time. Source: Center for Tobacco Policy Research

13 SUSTAINABILITY COMPONENTS Funding Stability Public Health Impacts Political Support Communications Strategic Planning Partnerships Surveillance & Evaluation Program Improvement Organizational Capacity Source: Center for Tobacco Policy Research

14 WHY EVALUATION? An essential element of program sustainability Continuously ask questions that matter Use evaluation findings for decision-making and action If you don t systematically collect information on what your program is doing you will not know how to communicate what is working and why it is working, what improvements need to be made, or how to make the case for future funding. P re s k ill H., & B oy le S. ( 2008). A m u lt idis c ipl in a ry m odel of e va luation c a p a c it y b u ilding. A m e r ic a n J o u r n a l o f Eva lu a t io n, 29,

15 QUESTIONS?

16 MAKE IT SIMPLE Identifying how to implement evaluation in your everyday practice

17 POLL What is your evaluation experience? Beginner Intermediate Advanced

18 WHERE DO I FIND AN EVALUATOR?

19 Source: University of Wisconsin-Extension Publications

20 EVALUATION PLAN TEMPLATE Evaluation Question Indicator/ Measure Data Source Data Collection Method Schedule/ Timeline

21 GENERATING AN EVALUATION PLAN Create a logic model to describe the program Identify stakeholders and brainstorm the evaluation purpose and what you want to know/learn about the program? (Engage) Write questions that reflect what you want to know? (Focus) Be specific and clear Who, What, When Where, How, and Why Ask, How will this information be used? Carefully consider what information is needed in order to answer each question. (Data Source) Consider how you can collect the information needed and in what format you want the information to be presented. (Data Collection Methods) Simultaneously consider the use of the information Prepare a plan for how the information will be complied analyzed, and interpreted. (Analysis) Prepare a plan for how the information will be formatted and disseminated (Use)

22 BASIC LOGIC MODEL A visual representation of the program. It shows what is put into the program, what you intend to do, what is expected to result from the program activities, and what is intend to change over the short, intermediate, and long term. Inputs Activities Outputs Short-term Outcomes Intermediate Outcomes Long-term Outcomes Process Evaluation Outcome Evaluation

23 SCHOOL/COMMUNITY GARDEN EVALUATION The Health in Action Coalition has to report to their funder how they used $1,000 to help provide seniors at the Georgetown Senior Center with an environment supportive of healthy living.

24 DEVELOPING EVALUATION QUESTIONS What does the funder want to know? How much did it cost to implement a school/community garden? What volunteers/partners engaged in the garden planning and implementation? To what extent is the garden integrated in the school curriculum/senior citizen activities? What challenges/barriers were encountered during the planning and implementation of the garden?

25 DETERMINING AND DEVELOPING EVALUATION MEASURES AND/OR INDICATORS Evaluation Questions (What do you want to know?) How much did it cost to implement a school/community garden? What volunteers/partners engaged in the garden planning and implementation? To what extent is the garden integrated in the school curriculum/community activities? What challenges/barriers were encountered during the planning and implementation of the garden and how were they resolved? Measures/Indicators (Evidence) - Cost of supplies - Volunteer in-kind hours donated - Partners - Staff - Volunteers - Students - Sustainability plan for garden upkeep maintenance - # Classrooms/subjects/grades that use the garden in the curriculum - Challenges - Solutions

26 IDENTIFYING DATA SOURCES Evaluation Questions (What do you want to know?) Measures/Indicators (Evidence) Data Source (Who has this information?) How much did it cost to implement a school/community garden? What volunteers/partners engaged in the garden planning and implementation? To what extent is the garden integrated in the school curriculum/community activities? What challenges/barriers were encountered during the planning and implementation of the garden and how were they resolved? - Cost of supplies - Volunteer in-kind hours donated - Partners - Staff - Volunteers - Students - Sustainability plan for garden upkeep maintenance - # of Classrooms/subjects/ grades that use the garden in the curriculum - Challenges/barriers encountered - Work around solutions Garden Coordinator Garden Coordinator - Community members - Teachers/Principles Garden Coordinator

27 DATA COLLECTION METHODS Evaluation Questions (What do you want to know?) Measures/Indicators (Evidence) Data Source (Who has this information) Data Collection Methods How much did it cost to implement a school/community garden? - Cost of supplies - Volunteer in-kind hours donated Garden Coordinator - Spreadsheet tracking expenses - Volunteer hours sign in sheet What volunteers/partners engaged in the garden planning and implementation? - Partners - Staff - Volunteers - Students Garden Coordinator - Volunteer sign-in sheet - Partner List/Log To what extent is the garden integrated in the school curriculum/community activities? - Sustainability plan for garden upkeep maintenance - # of Classrooms/subjects/ grades that use the garden in the curriculum - Community members - Teachers/ Principles - Review of Community sustainability plan - Review of classroom lesson plans/curriculum - Classroom observations What challenges/barriers were encountered.. - Challenges/barriers encountered - Work around solutions Garden Coordinator - Review of Work/Plan status column

28 POLL What type of evaluation design do the evaluation questions and data collection methods suggest? Quasi-Experimental Experimental Random Summative Non-Experimental

29 WALKING TRAIL EVALUATION ESMM Forest Acres has been funded to implement a community walking trail. They want to ensure that the walking trail is implemented as planned and document how the resources are used for this project. They hope to document lessons learned during planning and implementation in order to provide justification of an effective implementation model to future funders.

30 DEVELOPING EVALUATION QUESTIONS What do you want to know? Who was involved in the development of the walking trail? What was the process for getting the walking trail developed in Forest Acres? What challenges/barriers were encountered during the planning and implementation of the walking trail? How is the walking trail being advertised/promoted in the community?

31 POLL What type of evaluation does this scenario suggest? Impact Evaluation Performance Management Process Evaluation Outcome Evaluation All the above

32 DETERMINING AND DEVELOPING EVALUATION MEASURES AND/OR INDICATORS Evaluation Questions (What do you want to know?) What assessments or audits were performed to establish the need? What resources (financial) were necessary to implement the program? Who was involved in the development of the walking trail? What was the process for getting the walking trail developed in Forest Acres? To what extent was the walking trail implemented as planned? To what extent is the walking trail being advertised/promoted in the community? Measures/Indicators (Evidence) - Type of assessment(s) used - Amount of money spent on program planning and implementation - Partners - Staff - Volunteers Work plan listing specific activities conducted and successfully completed. Work plan documenting challenges/barriers with completing a specific work plan activities. Mechanisms/outlets/venues/ used to promote the trail

33 IDENTIFYING DATA SOURCES Evaluation Questions (What do you want to know?) What assessments or audits were performed to establish the need? What resources (financial) were necessary to implement the program? What was the process for getting the walking trail developed in Forest Acres? To what extent, was the walking trail implemented as planned? Measures/Indicators (Evidence) - Type of assessment(s) used - Amount of money spent on program planning and implementation - Tasks implemented to implement the walking - Challenges/barriers encountered during the planning and implementation of the walking trail Data Source (Who has this information?) - Walking Trail Coordinator - Report of the assessment findings Walking Trail Coordinator Walking Trail Coordinator Walking Trail Coordinator

34 DATA COLLECTION METHODS Evaluation Questions (What do you want to know?) Measures/Indicators (Evidence) Data Source (Who has this information) Data Collection Methods What assessments or audits were performed to establish the need? - Type of assessment(s) used - Walking Trail Coordinator - Report of the assessment findings - Document Review What resources (financial) were necessary to implement the program? - Amount of money spent on program planning and implementation Walking Trail Coordinator - Spreadsheet tracking expenses What was the process for getting the walking trail developed in Forest Acres? - Tasks implemented to implement the walking Walking Trail Coordinator - Project Work/Action Plan To what extent, was the walking trail implemented as planned? - Challenges/barriers encountered during the planning and implementation of the walking trail Walking Trail Coordinator - Review of the Project Work/Action Plan

35 WHAT TO CONSIDER WHEN SELECTING DATA COLLECTION METHODS Evaluation Questions Evaluator skills/experience Resources Stakeholder preference Level of acceptable intrusiveness Accuracy of Information Availability Timeliness Funding requirements

36 DATA COLLECTION METHODS Questionnaires, surveys, checklists Interviews Document review Observation Focus Groups Expert or peer review Photographs/videos Logs Community Forum

37 DATA ANALYSIS Not all evaluation collection methods require formal analysis. Quantitative data analysis Calculate basic descriptive statistics (frequency, mean, median, mode) Additional Statistical Analysis correlations, chi square, t-test, and analysis of the variance Qualitative data analysis Content analysis Pre-set or emergent categories Multiple reviews to make meaning of the data

38 PLANNING FOR DATA ANALYSIS It is important to plan for how data will be analyzed before it is collected. Consider: What data analysis method (i.e. frequency, mean etc ) would be appropriate and why? Will this data analysis method (i.e. frequency, mean etc ) answer my question? What do you hope to learn by analyzing the data using this method?

39 DATA REPORTING Quantitative Number of respondents Data tables to that include frequency, percentage, mean etc Chart/Graphs Qualitative Number of people who provided comments Reoccurring themes (provide quotes) Unexpected findings Ensure quotes remain anonymous unless permission is obtained in advance

40 COMMUNICATING THE EVALUATION PROCESS AND FINDINGS Report how you conducted the evaluation and what was discovered Not all evaluations will result in a formalized written report. The format in which the evaluation findings are packaged should be based on the primary intended users and uses. Group discussions Executive summaries Newsletter, bulletin, brief, brochures Video/Media communication

41 RESOURCES University of Wisconsin-Extension Publications Patton, M.Q. (2008) Utilization-focused Evaluation. 4 th edition. Thousand Oaks, CA: Sage Publications.

42 QUESTIONS?

43 MAKE IT HAPPEN How putting program monitoring data into SCORES helps monitor the implementation of the Options for Action (SC Obesity State Plan)

44 PUTTING IT ALL TOGETHER Change happens at the local level Building local level capacity to conduct effective program monitoring and evaluation will enable South Carolina to better understand What is going on to address obesity? What strategies works and why? What strategies do not work and why? Partners Best Practice Implmtn Evaluation Change

45 QUESTION How do we show what the state of SC is doing to make healthy eating and active living essential to the everyday culture where we live, work, learn, pray, and play? Monitoring Evaluation

46 PROGRAM MONITORING Tracking, documenting, summarizing the inputs, activities, and outputs of the South Carolina Obesity State Plan Examples: Types of objectives/activities implemented to work toward policy and environmental change Policy adoption Environmental changes

47 POLL Have you used SCORES to input your coalition/organization accomplishments? Yes No

48 SOUTH CAROLINA ONLINE REPORTING & EVALUATION SYSTEM Centralized online reporting system used to document how the South Carolina Obesity State Plan (Options for Action) Objectives and Activities are being implemented by community and state level partners Consistent framework to guide reporting of nutrition, physical activity, and obesity prevention efforts

49 SCORES IN LOGIC MODEL TERMS Partners Funding Tools Inputs Activities Objectives Activities Action Types # of People Reached Resources Leveraged Outputs Short Term Policy & Environmental Change Implementation Evaluation Outcome Evaluation Inter-Term Long Term Advanced Evaluation Primary Focus of SCORES

50 OBESITY STATE PLAN EVALUATION Evaluation Question Indicators/Measures Sources (who has this information?) Data Collection Methods Schedule/ Timeline What has occurred on the local level to address policy, systems, or environmental change related to nutrition, physical activity, and obesity? - # of OFA objectives being implemented - # and type of local level policy changes adopted - # and type of local level environ. changes adopted - ESMMSC Chapters - Local coalitions - SCDHEC Region Staff - Working Well Centers of Excellence - Partners - Review of SCORES data - Quarterly

51 WHY ENTER INTO SCORES? To help develop models of implementation so that other local coalitions can replicate what worked well To demonstrate what your local community is doing to impact the burden of obesity in South Carolina To make contributions towards County Profile Sheets

52 WHAT CAN BE LEARNED FROM Enables the state of South Carolina to answer the following questions: Who is implementing Options for Action? What is being implementing? Where is implementation occurring? What policies and environmental changes are occurring? What partners are supporting implementation of nutrition, PA, and obesity prevention efforts? How much money is being leveraged? What challenges/barriers are being encountered during implementation of OFA Activities? And many more. SCORES DATA?

53 SUMMARY Make It Plain Purposeful and systematic process of understanding what you do. Make it Useful Evaluation will help sustain your efforts. Make It Simple Ask questions and outline the process for how to answer those questions. Make It Happen Evaluation occurring at the local level will pave the way for evaluation at the state level.

54 QUESTIONS Kristian L. Gordon Evaluation Coordinator

55 SURVEY MONKEY 4 brief questions