1. KEY CONCEPTS IN EVALUATION: WHAT EVERYONE MUST KNOW

Transcription

1 1. KEY CONCEPTS IN EVALUATION: WHAT EVERYONE MUST KNOW Important Definitions Clarifications About Evaluation Strategies Evaluation is a science with a relatively short history. It became a distinctive field of professional social science practice in the late 1960 s (Patton 1982). There are many types and classifications of evaluation, and there are many terms associated with the practice (see glossary in the appendix). This guidebook is focused on participatory program evaluation. The following are key definitions and points of clarification necessary to orient the remaining sections of this manual. Working Definition of Program Evaluation. The practice of evaluation involves the thoughtful, systematic collection and analysis of information about the activities, characteristics and outcomes of programs, for use by specific people, to reduce uncertainties, improve effectiveness, and make decisions regarding those programs (adapted from Patton 1982). Working Definition of Participatory Evaluation. Participatory Evaluation involves trained evaluation personnel and practice-based decision-makers (i.e., organizational members with program responsibility service providers) working in partnership. It brings together seasoned evaluators with seasoned program staff to provide training, and then design, conduct, and use results of a program evaluation (Cousins, 1998). There is additional information in the complementary Participatory Evaluation Essentials A Guide For Non-Profit Organizations And Their Evaluation Partners about participatory evaluation, how to commission external evaluation and/or evaluation consultation. Many years of participatory evaluation practice show that program providers, funders and evaluators can all be meaningfully engaged in program evaluation. It is important to recognize the following. All evaluations are partly social (because they involve human beings), partly political (because knowledge is power), and only partly technical (Herman, Morris, Fitz-Gibbons, 1996). Evaluation data can be collected using qualitative methods (e.g., observations, interviews) and/or quantitative methods (e.g., surveys, statistical analyses of practical assessments). Although there has been much debate about which strategies and types of data are best, current thinking indicates that both are valuable and both can be collected and analyzed rigorously. There are multiple ways to address most evaluation needs. Different evaluation needs call for different designs, types of data and data collection strategies. These Integrating Evaluative Thinking Bulletins provide a framework to help organizations and their partners think about ways to enhance evaluative thinking in multiple aspects of their organizations. Readers seeking additional information about evaluation are advised to visit the Evaluation Capacity Building section of the Effectiveness Initiative pages of the Bruner Foundation website, ( for more tools and resources. This bulletin was developed by Anita Baker with Beth Bruner to help organizations integrate evaluative thinking into their organizational practice. After a successful effort to help organizations build evaluation capacity (see for more information about REP ), the Bruner Foundation has been working to help organizations use that capacity in other management areas. The Foundation continues pursuing its strong belief that using evaluation skills, specifically identifying key questions of substance, determining what data are needed to answer questions, gathering appropriate data in systematic ways, analyzing data, sharing results, and developing strategies to act on evaluation findings, is valuable in everyday work of organizations. The contents of this bulletin were influenced by the non-profit organizations in Rochester that are former REP partners and ongoing consumers of these efforts, as well as by the work of Michael Quinn Patton, Paul Connolly, Paul Light, and Peter York, and the many others who thoughtfully add to the evaluation and organizational capacity-building literature.

2 What is Needed in an Evaluation Design? An evaluation design is a way of helping you think about and structure an evaluation. It communicates evaluation plans to evaluators, program officials, and other stakeholders. A good design should include the following. *summary information about the selected program and why it is to be evaluated (logic model summary where available see next section) * questions to be addressed by the evaluation * data collection strategies that will be used * individuals who will undertake the activities * schedule for data collection activities * products of the evaluation, who will receive them, and how they will be used * projected costs to do the evaluation (where appropriate) Don t forget, evaluation, even self-assessment is not free. But evaluation should not be viewed as in competition with program resources. Evaluations can be funded as a component of a program (using program funds) or as a separate project (using earmarked or additional funds). A common rule of thumb is to set aside 10-20% of the cost of the program, for evaluation. Purpose of Evaluation Program evaluations are typically conducted to accomplish one, two or all of the following. ** To Render Judgments ** To Facilitate Improvements ** To Generate Knowledge It is critical to carefully specify why evaluation is being conducted. This should be done at the earliest stages of evaluation planning and with the input of multiple stakeholders (e.g., staff, board, participants). Questions Get you Started! Evaluation questions should ideally be determined by the service providers together with the evaluator, in accordance with the purpose of the evaluation. Evaluation questions. focus and drive the evaluation (they clarify what will and will not be evaluated). should be carefully specified (and agreed upon) in advance of other evaluation design work. generally represent a critical subset of information that is desired about a program to address the purpose of the evaluation. What are the Criteria of Good Evaluation Questions? It is important to keep the number of evaluation questions manageable. The exact number of evaluation questions depends on the purpose of the evaluation and resources available to conduct the evaluation, but limiting the evaluation to address between two and five questions is strongly advised. The following are criteria of good evaluation questions (adapted from Patton, 1997): It is possible to obtain data to address the questions. (The data must be available for use though, to those undertaking evaluation.) There is more than one possible Aanswer@ to the question, i.e, the findings are not predetermined by the phrasing of the question. Those conducting the evaluation want and need information to help them address the questions, and know how it will be used internally and where appropriate externally. The questions are aimed at changeable aspects of programmatic activity (i.e., they should focus on those things which can be modified where findings warrant change). Page 2

3 Using Logic Models to Inform Evaluation Design A Logic Model is a simple description of how a program is understood to work to achieve outcomes for participants. Developing a logic model helps you to identify your vision, the rationale behind your program, and how your program will work. The following shows a format for a simple logic model. (Full-sized forms for reproduction are included in the appendix to the Guidebook for Nonprofit Organizations and their Evaluation Partners) Inputs Activities Outcomes You can choose to... * Embed targets Adapted from the United Way of America *Distinguish between initial, intermediate and longer term. Mark those to be tracked or measured. Why Use a Logic Model? Developing a Logic Model will help you get clear about what you re doing, and how you hope it will make a difference. You have the best knowledge of the context of your work and what s important to you and your communities. Developing a Logic Model draws from what you already know. A Logic Model will leave you with a clear, thoughtful plan for what you are doing and what you hope to achieve. This plan can be an advocacy resource, bring clarity to your message and help you tell your story. Summarizing Logic Models helps inform evaluation planning. A good program logic model... clearly communicates what is to be done and how it is expected to help participants seems logical suggests appropriate connections between inputs, activities and outcomes has outcomes that are relevant to the mission/objectives of the program, represent meaningful change for participants and are likely to be accepted as important by program managers and other stakeholders has activities sufficient in number, duration and intensity to contribute significantly to the outcomes but reasonable given project inputs includes a comprehensive set of activities that are related to the outcomes has inputs that are sufficient (in both quantity and quality) to support all activities

4 The Importance of Evaluation Stakeholders Evaluation stakeholders are people who have a vested interest in evaluation findings. (Patton, 1997, p.41). Stakeholders include anyone who makes decisions about a program, desires information about a program, and/or is involved directly or indirectly with a program. Most programs have multiple stakeholders. Stakeholders typically have diverse and often competing interests. In program evaluation, stakeholders typically include organizational officials (eg., Executive Director), program staff, program clients or their caregivers, and program funders. Sometimes community members or other organizations are also stakeholders. It is critical for providers and evaluators to involve some key stakeholders in the evaluation process, especially during planning. There are also usually important roles for stakeholders in data collection, analysis and reporting. How are Evaluation Data Collected? There are four primary types of evaluation methodologies/data collection strategies which can (and should) be combined to address evaluation questions and allow for multiple sources of data. This includes: Record/Document Review, Surveys, Interviews, Observation. All have both benefits and limitations and require preparation on the front end. **Remember, mixed methodologies, and multiple sources of data/respondents, collected at multiple points in time, increase evaluation rigor and usefulness of findings. **Remember, data do not have to be collected for all participants in every program cycle. Evaluating findings from samples (subgroups) of participants or point-in-time (snapshot) estimates can be a useful approach. What Happens after Data are Collected? Once you have collected evaluation data, there are three important stages left: 1) You must analyze all your data and summarize the results. Organizations are great at collecting data, but they often forget to analyze and review the results that MUST be done to obtain results to inform decision-making. 2) You have to convert your findings into a format that can be shared with others. Sometimes this is a brief memo with key findings, sometimes an oral report, but the most common form used to summarize evaluation results is the Evaluation Report. (See Sections IV and V for more details about analyzing data and reporting findings). 3) You have to develop action steps from your findings. This is probably the most important step it moves evaluation from perfunctory compliance into the realm of usefulness. Developing action steps from evaluation findings is a key component of Evaluative Thinking organizations that regularly use Evaluative Thinking always make it to this stage. Action Steps complete this statement: Now that we know we will do the following. Page 4

5 How to Use These Bulletins The Integrating Evaluative Thinking Bulletins were developed in response to the continuing need expressed by nonprofit partners to clarify how to enhance evaluative thinking and put enhanced evaluation capacity to work. The bulletins are intended to do the following: Clarify what evaluative thinking is, why it is important, and who can do it. Provide direction regarding the use of the Bruner Foundation s Evaluative Thinking Assessment Tool to inform action plans that will enhance evaluative thinking. Describe additional ways that evaluation strategies can be used internally to increase evaluative thinking (e.g., evaluating the effectiveness of staff development efforts or collaborative ventures). Help organizations think about creative ways to present and use evaluation findings. Provide practical advice on how to hone analytical skills, and use Evaluative Thinking when making decisions about standard organizational procedures including governance, mission development, strategic planning, fund development, leadership, technology, human relations, staff development, alliances and business ventures. There are 11 individual Integrating Evaluative Thinking Bulletins including this edition. They cover the following topics: evaluation basics and definitions, evaluative thinking basics and assessment of evaluative thinking, evaluation and nonprofit boards, commissioning evaluation, collecting, analyzing and using evaluation data, communicating about evaluation, evaluation and technology, evaluation and HR, evaluation and alliances, increasing participation in evaluation and sustaining evaluative thinking. Each bulletin is brief and replete with practical suggestions made by nonprofit partners who reviewed the work (including some very specific pointers highlighted in yellow). A complete set of all Bulletins, as well as other complementary tools and resources are available via the Bruner Foundation website, We encourage all users of the bulletins to: Familiarize (or re-familiarize) yourself with basic information about evaluation capacity. Orient yourself regarding the Bruner Foundation s Evaluative Thinking Assessment Tool and conduct preliminary assessment in your organization. Develop action plans informed by the suggestions found in the bulletins. Implement action plans to enhance Evaluative Thinking in your organization. Use these specific bulletins as reference materials when needed. Visit the Bruner Foundation website Effectiveness Initiatives pages ( to familiarize yourself with the history behind this effort and to access the tools and resources available there. ** These materials are for the benefit of any organization and may be used in whole or part provided that credit be given to the Bruner Foundation. They may NOT be sold or redistributed in whole or part for a profit.