Are You Ready? Assessing Your Organization s Readiness to Conduct Evaluation

Transcription

1 Are You Ready? Assessing Your Organization s Readiness to Conduct Evaluation Colleen F. Manning, M.A. and Irene F. Goodman, Ed.D. Goodman Research Group, Inc. April 2016 This article builds on a breakout session we presented at the NSPA 2015 conference in Charleston, as part of a conference strand on evaluation. The session was well attended, primarily by scholarship program directors and managers, but also by Foundation presidents and program officers. We have written this article with a similar audience in mind. What is evaluation readiness? And, why does it matter? Scholarship program directors play an important role in developing and processing scholarship applications; outreach and marketing; the scholar selection process; building relationships with donors, colleges/universities, and high schools; student outreach and communication; and database creation and management. Program directors are also increasingly accountable to a wide range of stakeholders, which has motivated their interest in evaluation. In addition to wanting to demonstrate program outcomes to stakeholders and substantiate requests for increased funding (Pell Institute Evaluation Toolkit), scholarship providers also have intrinsic motivations for evaluation. Evaluation produces programmatic selfunderstanding, identifies program strengths that can be leveraged, exposes ineffective practices that can be modified, and creates a foundation for strategic planning (Pell Institute Evaluation Toolkit). Balancing external accountability with these internal reasons for evaluation is a key challenge in nonprofit human service evaluation (Alaimo, 2008). Program directors interested in evaluation can play a crucial part in helping to ready their organizations for evaluation. What do we mean by evaluation readiness? And, why does it matter? Innovation Network (2012) defines evaluation readiness as a term used to describe an organization or program s ability to successfully implement an evaluation project or framework. Readiness is important whether an organization is conducting an evaluation internally or engaging an external evaluator. Other closely related terms you may hear bandied about include evaluation capacity building and evaluability. In addition to an organization s ability to successfully implement evaluation projects, evaluation capacity building (ECB) also emphasizes cultivating organizational support for program evaluation as well as overall organizational learning (Taylor-Powell & Boyd, 2007). Evaluability is an attribute of a program and refers to the program s ability or readiness to be meaningfully evaluated, in a reliable and credible manner, so as to contribute to improved program performance and management (Davies, 2013; Juvenile Justice Evaluation Center, 2003).

2 In this article, we introduce you to a diagnostic tool to assess your organization's and scholarship program s readiness for evaluation and make some suggestions for action planning based on the results of the tool. But let s start with some key messages about getting ready for evaluation: First, an organization doesn t become ready to incorporate evaluation overnight. It s a process. Second, readiness is not black and white; it s not that an organization is ready or it s not. Rather, organizations have different strengths to leverage and areas to develop when it comes to being ready for evaluation. Third, it s important not only to build capacity to conduct evaluation but to build capacity to use and learn from evaluation. A key ingredient of evaluation capacity is evaluative thinking. By that we mean being reflective, asking substantive questions, and believing in the value of data (Bruner Foundation, 2007). This is closely related to organizational learning, a continuous process of organizational growth and improvement (Preskill & Torres, 2000). For another activity that helps promote organizational learning and evaluative thinking, see our companion article, Developing a Logic Model for Your Scholarship Program. Fourth, there are multiple components to evaluation readiness. Broadly speaking, we can assess organizational evaluation capacity and we can assess the evaluability of a given program within an organization. Both tell us whether program evaluation is warranted, realistic, and apt to be useful. Assessing Your Organization s Evaluation Readiness A number of tools have been created to help organizations assess and strengthen their readiness for incorporating evaluation and learning into their processes and routines (Bruner Foundation, 2007; Capacity for Health; Innovation Network, 2012; Preskill & Torres, 2000; Volkov & King, 2007). We suspect that many such tools have been inspired by the Organizational Capacity Assessment Tool (OCAT) created by McKinsey and Company for Venture Philanthropy Partners. We encourage you to take a look at the OCAT if your organization is interested in an assessment of overall organizational capacity. Based on a review of these tools and the broader literature, we have designed a tool with elements we believe are most relevant for scholarship-providing organizations. A copy of the tool is provided at the end of this article. A total of 31 capacity elements are clustered into four broad dimensions of evaluation capacity: 1. Organizational Context & Culture 2. Leadership

3 3. Program Readiness which is segmented into Program Design, Program Implementation, and Capacity to Provide Data 4. Evaluation Planning We suggest that a group of people representing the key functions or departments within your organization first complete the tool individually. (You can disseminate the tool individually or convene a group for this purpose.) This group should include your organization's executive director/ceo and board president as well as other staff, board members, and/or important constituents. We believe it is important that participants complete the tool anonymously (i.e., no names) to help ensure the most honest responses. Instructions Emphasize to participants that there are no right or wrong answers and that everyone is likely to have different views regarding organizational capacity. Instruct participants to respond based on their own experiences and not on how they think other group members would respond (or would want them to respond). Tell participants to respond based on what actually happens in the organization and not on the organization s desired state. To complete the tool, for each element of capacity, respondents should read the statement and then circle the number that best represents their opinion about the organization s capacity: a 1 if they Disagree with the statement; a 2 if they Slightly Agree with the statement, a 3 if they Somewhat Agree with the statement; and a 4 if they Strongly Agree with the statement. We ve also included a notes column, for aha moments and questions. If a participant is trying to decide between two answers, it is advisable to underestimate rather than overestimate capacity. Analyzing and Interpreting the Results This tool is designed to spark dialogue. For that reason, we recommend analyzing and interpreting the results as a group. Participants responses can be entered into an Excel spreadsheet ahead of an in-person meeting or tallied by hand at the meeting. Either way, we suggest creating some kind of graphic display of the results, as visual aids often inspire action planning (Bruner Foundation, 2007). There are three ways you can examine your results: 1. First, you can calculate an average for each of the 31 individual items to see very specific areas for improvement (i.e., items with an average of 1 or a clear need for increased capacity) and very specific areas of strength (i.e., items with an average of 4 or a high level of capacity in place). 2. Second, using the row totals, you can calculate an average for each of the seven broad dimensions of evaluation capacity Organizational Context & Culture, Leadership, Program Readiness Program Design, Program Readiness Program Implementation, Program Readiness Capacity to Provide Data, and Evaluation Planning to see general

4 areas for improvement and general areas of strength. We ve included a rubric below; however, your organization may decide on different thresholds. Organizational Context & Culture Leadership Program Design Program Implementation Program Capacity to Provide Data Evaluation Planning Clear need for increased capacity Basic level of capacity in place Moderate level of capacity in place High level of capacity in place Lastly, by averaging the grand totals (the last row of the tool), you can look at the overall result across all items and areas for a very high-level sense of your organization s evaluation readiness. The minimum grand total (indicating the lowest possible capacity) would be 31 while the maximum grand total (indicating the highest possible capacity) would be 124. The process of analyzing and interpreting the results is an opportunity to facilitate discussion about different perspectives on capacity. It is an opportunity for members of the organization to see where they agree and disagree and to learn from one another. Your organization may wish to discuss ratings until a consensus is reached on one set of ratings that best represents the organization. There are those who believe this team approach improves validity and reduces individual biases (Massachusetts Cultural Council). However, keep in mind that this tool is not designed as a definitive measurement tool. Further, it may be very difficult to reconcile widely disparate ratings. What next? We recommend using the results of the Evaluation Readiness Tool to identify tangible next steps that your organization can take to strengthen your readiness for evaluation and learning. The results may point to numerous potential steps, and you may need to set priorities. In vetting your action steps, think about potential barriers and ways to address those barriers (Capacity for Health) and consider the resources that may be necessary to support potential action steps (Innovation Network, 2012).

5 Evaluation Readiness Tool Organizational Context & Culture Disagree (suggests clear need for increased capacity) Slightly Agree (suggests basic level of capacity in place) Somewhat Agree (suggests moderate level of capacity in place) Strongly Agree (suggests high level of capacity in place) 1. Our organization builds in time for individual and group reflection about services, products, and processes Organizational staff regularly engage in knowledge sharing and transfer Employees continuously look for ways to experiment and innovate to improve services, products, and processes The internal environment of our organization is supportive of change Programming decisions are based in part on program evaluation results, or on data and evidence There is general commitment to evaluation throughout our organization Information is regularly gathered from stakeholders to gauge organizational and programmatic strengths and weaknesses Currently available information tells us what we need to know about the effectiveness of our programs, processes, products, and services Stakeholders/funders are expecting our organization to report on evaluation results. 2 Leadership 10. Leaders walk the talk and demonstrate commitment to evaluation, organizational learning, and evidence-based decision-making Leaders educate staff about the value of evaluation Leaders devote necessary resources/time for evaluation and learning. 4 Program Readiness Program Design 13. The program s long-term impact and outcomes are clearly identified The intended beneficiary group is clearly identified The program has well-defined objectives Program goals and objectives are related to program activities and are realistic and attainable. 5 Program Readiness Program Implementation 17. The program serves its intended population (e.g., students in a particular county or field of study) The program has the required resources (e.g., sufficient Aha! moments & questions

6 number of staff, necessary equipment) The program activities are being implemented as designed. 5 Program Readiness Capacity to Provide Data for an Evaluation 20. The program uses forms to document services provided and Scholar activities/participation The program has developed or adopted measures to assess progress in achieving goals and objectives The program has in place or can develop procedures to generate data needed for an evaluation. 5 Evaluation Planning 23. Our organization convenes meetings to discuss evaluation planning Our organization develops and follows an evaluation plan (i.e., that clarifies what is to be evaluated and how) There are organizational funds dedicated to evaluation There is a key staff person (or persons), with evaluation expertise, to address our organization s evaluation needs Our organization hires evaluation consultants when needed Sufficient information is available on how to access existing evaluation resources (e.g., websites, professional organizations, evaluation consultants) There are structures and systems in place to systematically gather, store, analyze, and use data We currently use (or plan to use) evaluation data outside our organization to increase funding and share promising practices Primary users of an evaluation have been clearly identified. 3 GRAND TOTAL This tool was created drawing from seven existing organizational capacity assessment tools (or related articles). Individual questions are drawn or adapted from each: 1) Bruner Foundation, Modified Evaluative Thinking Assessment Tool. 2) Capacity for Health. Monitoring and Evaluation (M&E) Capacity Assessment Tool. 3) Davies, R., Planning Evaluability Assessments: A Synthesis of the Literature with Recommendations. 4) Innovation Network, Capacity and Organizational Readiness for Evaluation (Core) Tool. 5) Justice Research and Statistics Association, Evaluability Assessment: Examining the Readiness of a Program for Evaluation. 6) Preskill, H. & Torres, R. T., The Readiness for Organizational Learning and Evaluation Instrument (ROLE). 7) Volkov, B. B. & King, J. A., A Checklist for Building Organizational Evaluation Capacity.

7 References Alaimo, S. P. (2008). Nonprofits and evaluation: Managing expectations from the leader s perspective. In J. G. Carman & K. A. Fredericks (Eds.). Nonprofits and evaluation. New Directions for Evaluation, 119, Bruner Foundation (2007). Modified Evaluative Thinking Assessment Tool. Retrieved from Capacity for Health. Monitoring and Evaluation (M&E) Capacity Assessment Tool. Retrieved from Capacity-Assessment-Tool3.pdf Davies, R. (2013). Planning evaluability assessments: A synthesis of the literature with recommendations. Department for International Development. Working Paper 40. Retrieved from -planning-eval-assessments.pdf McKinsey and Company for Venture Philanthropy Partners. (2001). Organizational Capacity Assessment Tool. Retrieved from Innovation Network. (2012). Capacity and Organizational Readiness for Evaluation (Core) Tool. Retrieved from Justice Research and Statistics Association. (2003). Evaluability Assessment: Examining the Readiness of a Program for Evaluation. Retrieved from Massachusetts Cultural Council Cultural. Investment Portfolio Organizational Capacity Measurement Tool. Retrieved from Pell Institute. Evaluation Toolkit. Retrieved from Taylor-Powell, E., & Boyd, H. H. (2008). Evaluation capacity building in complex organizations. In M. T. Braverman, M. Engle, M. E. Arnold, & R. A. Rennekamp (Eds.), Program evaluation in a complex organizational system: Lessons from Cooperative Extension. New Directions for Evaluation, 120,

8 Preskill, H., & Torres, R. T. (2000). The Readiness for Organizational Learning and Evaluation Instrument (ROLE). Retrieved from Volkov, B. B., & King, J. A. (2007). A Checklist for Building Organizational Evaluation Capacity. Retrieved from pdf