The Effectiveness of Evaluation in the Social arena in Israel

Transcription

1 The Effectiveness of Evaluation in the Social arena in Israel Perceptions and uses of evaluation research by agencies funding social programs in Israel Abstract This research was carried out with the support of the Rothschild Caesarea Foundation Research Team: Dr. Lior Rosenberg Program Evaluator Tirzah Margolin Program Evaluator Anat Kedem Director, Program Evaluation Unit Dr. Tali Freund VP Assessment and Evaluation April 2014 The Center For Educational Technology (CET), N.G.O, 16 Klausner St. Tel Aviv, Tel: fax: ,

2 2015 The Center for Educational Technology Assessment and Evaluation Department 16 Klausner St. P.O.B Tel Aviv Israel The Center for Educational Technology (CET) is an independent, nonprofit organization established in 1971 by the Rothschild Foundation. CET is dedicated to the advancement of education in Israel and in Jewish communities abroad by providing teaching materials and learning environments that combine quality content, rich technology and advanced pedagogy. CET's Assessment and Evaluation Department offers state-of-the-art tools and services supporting intervention agencies in program planning, monitoring, outcome and impact evaluation in the fields of education, community and social welfare. The department also develops a wide variety of national achievement exams, including computer-based ones. 1

3 "For us evaluation is an outside resource. A compass that examines if we are moving in the right path to reach the impact and goals we want to achieve. In addition, evaluation plays a role in conceptualization. The partnership with the evaluators helps us in identifying our goals and see if they can be measured. This process has a strategic importance for us" "Evaluation is an amazing tool for decision making regarding the design of a project and its financial support. A project that presents data and outcomes is not left from the discussion and is highly considered when financial support is on the table. Data has a huge impact" 2

4 Research Questions The present evaluation research deals with the way and extent to which evaluation research influences the decisions of funding agencies in the social arena in Israel. There were four key questions underlying the research: 1. To what extent do the evaluation research data serve the funding agencies in making decisions and shaping strategies regarding intervention programs in the social arena? 2. Is it possible to characterize different types of funding agencies based on the extent and manner of their use of the evaluation research findings? 3. Is it possible to characterize aspects of the evaluation process, which influence the scope and manner of use of the evaluation reports by the funding agencies? 4. What characterizes the outputs of evaluation research that are used more extensively by the funding agencies? The full evaluation report presents a literacy review that includes global studies regarding the use of evaluation by funders. The review presents the role of the three funding agencies in the social arena in Israel: government, municipal authorities and NGOs and the way those agencies use evaluation as well as the impact of evaluation. Methodology The evaluation research participants include representatives of government (including government authorities and services), municipal authorities and NGOs (foundations and Jewish federations), involved in funding social initiatives in Israel. The evaluation research was carried out in two stages. In the first stage, 26 in-depth interviews were carried out, from January to May The findings from the interviews provided the basis for development of an online survey questionnaire, which was used for the second stage of the evaluation research. The questionnaire was sent to 250 representatives of funding agencies, between July and September In total, 79 respondents answered the survey, of whom 16 (20%) were government representatives; 25 (32%) were representatives of municipal authorities; 27 (34%) were representatives of foundations and 11 (14%) were representatives of Jewish federations. Almost all of the respondents were in high rank positions (98%). 3

5 Main Findings The majority of funders who responded to the survey (83%) reported that their organization had initiated evaluation research, in the last three years. This finding may indicate that response to the survey was higher among those in favor of evaluation practices and may suggest an overestimation of the extent to which evaluation research is conducted for funding agencies. Findings indicate that in government departments and NGOs, there is a greater tendency to employ experts in the field of evaluation on their payroll, while municipal authorities tends to use external evaluation services (90% vs. 68%). The findings indicate that the funding agencies that initiated evaluation research report a high level of involvement in most of the stages of the evaluation process, although involvement is relatively limited at the stage of writing the tender for the evaluation research, and at the stage of drawing conclusions and setting recommendations. Chart 1: The involvement of funding agencies in the various stages of evaluation 1 1 The chart reflects the percentages of responders who reported that they were involved in each of the specified evaluation stages (on a Yes/No scale). 4

6 Findings also point to the fact that the funding agencies which reported a greater involvement in the evaluation process also tended to use such research to evaluate more issues, and made greater use of the findings (92%). Findings show that evaluation is perceived as an effective practice among funding agencies particularly for the purpose of monitoring implementation of the program (71%) and measuring its outputs and influence (80%). A high proportion of the funders believe that evaluation research is an effective mean of bringing about improvement (73%), but much less an effective mean of broadening partnerships and raising funds (48%). Chart 2: The use of research findings by the funding agencies 2 2 The chart reflects the percentages of responders who reported that they use each of the options (on a Yes/No scale). 5

7 Evaluation research serves the funding agencies mainly for improving their programs (95%) and less for influencing the authorities' policies (45%), as part of their public relations strategy, and/or as leverage for fund raising (43%). Similarly, it is seldom used for the purpose of comparing and examining alternatives to the current programs (40%), or for considering suspension of programs, or expanding or reducing investments in them (29%). Funding agencies seem to have a traditional perception of evaluation research, rather than seeing it as a resource that would assist in strategy building and decision making regarding their support and funding. Two major weaknesses of evaluation research were indicated by the survey's respondents: 1. Research reports usually lack a comparison of the programs under study with similar programs in Israel and abroad, or with other benchmarks; 2. Research reports lack compelling conclusions and recommendations. Along with these two main weaknesses, consideration should also be given to further faults, which were mentioned by the interviewees, and were indicated by more than a quarter of the survey's respondents; among them: the evaluation research findings are not delivered in accordance with the timing of decisions; the evaluation research reports are too long, and contain too much statistical data; and, they are not accompanied by a useful abstract. Chart 3: Barriers in using evaluation findings 3 3 The chart reflects the percentages of responders who reported that the specified barriers influence the use of the evaluation findings to a large extent or a very large extent (the scale ranged from "not at all" to "a very large extent"). 6

8 In regard to further development of the field of program evaluation, the majority of funders (59%) believe that it would be useful to set up an internet website that would provide comprehensive information on evaluation research and evaluators in Israel, to provide training to the agencies staff and/or to create communities of similar organizations around the use of evaluation research (55%), and last to set up a foundation that would fund research for NGO's (50%). When the funding agencies were asked whether they would be interested in several "nontraditional evaluation services, they showed a high interest in cost-effectiveness evaluation research (64%), in support and assistance for making effective use of the findings of evaluation research (61%). In addition they showed interest in evaluation research that will track impact and influences beyond the target beneficiaries and beyond the program s goals (56%). These findings indicate the potential for development of the program evaluation field, toward a more strategic and systemic direction. The perceptions and patterns of the use of evaluation, as reported by funding agencies from the three sectors, indicate the differences between them, deriving in our view from their position in the social arena government authorities, as the sovereign power, dictating policy and allocating resources, and subject to oversight and accountability; municipal authorities, being dependent on partnerships and fundraising to run services for their residents; foundations and Jewish federations, as the allocators of support to organizations that operate programs, and which are required to report to their donors. Alongside differences between the sectors, the research also identified obstacles to effective use of evaluation research, among them: a lack of finance or willingness to invest resources only in the program s direct beneficiaries (30%), along with a lack of familiarity with the field of evaluation among funding agencies (43%), which causes them to avoid it. A further major obstacle to obtaining the full effectiveness from evaluation research is the in-built tension between the funding agency and the operating organization, caused by the evaluation intervention. Insights and Recommendations The research indicates that there is broad agreement among the funding agencies that evaluation research is an effective means of bringing about improvement, and that the majority of the participants in the research indeed make such use of it. At the same time, the common perceptions and uses are traditional, reflecting an approach under which evaluation principally looks at program implementation and measure its outputs and influences, rather than as a strategic resource for shaping policy and decision making. These findings indicate that evaluation research has limited influence in the social sphere. 7

9 The challenge of turning evaluation research into a more meaningful tool for funding agencies, and reinforcing its contribution to social action in Israel, is therefore a key issue for discussion. The finding that funding agencies show interest in more strategic evaluation services, when these are offered to them directly, demonstrates a potential that is not yet fully exploited. We believe that perception of the evaluation activity as a process that can generate a strategic change is the key to increasing its effectiveness and influence in the social arena. In order to promote such a perception, we believe that the following are required: 1. Strengthening dialogue between evaluator and funder: An intensive, ongoing dialogue would be expected to promote recognition of the needs and limitations of each of the partners, and serve as a catalyst for change in perception and in implementation of the evaluation activity. Such a dialogue would contribute to the relevance of evaluation research and to its potential to bring about change. Placing such dialogue into the evaluation research program, in a structured way, will ensure its continuity, and it will then serve as an essential mechanism for motivating and managing change. Responsibility for this would be jointly shared by the funder and the evaluator. 2. Creation of a platform for management of change: Responsibility for the management of change lies with the funding agency, while the evaluators are supposed to assist them. It is very important that this aspect of the evaluation activity be discussed between the evaluator and the funder, and that they frame management mechanisms that will from the outset allow and support a process of change. 3. Change in perception of the evaluator s role: The idea that the work of the evaluators does not end with the delivery of findings, and that they are expected also to assist in the process of change, means a change of the evaluator s role perception. This change requires the evaluators to adopt a different understanding as to their function, to engage in a different kind of dialogue with the funder, and also requires the acquisition of consultative knowledge and skills, alongside their research skills. This may also call for working in a multidisciplinary evaluation team. 4. Expanding the knowledge among the evaluation clients, and creation of a shared language between evaluation client and evaluators: Without in-depth understanding among funding agencies of the potential and capabilities of the evaluation activity, it will lose its ability to become a leverage for change. Evaluation is a strategic management resource, and it is important that funding agencies have knowledge of how to exploit it fully and leverage it to promote the organization s goals, and that they consistently broaden their knowledge. At the same time, it is important for the evaluators to describe to the funder, in as much detail as possible, the evaluation services that they can provide; they should assist the funder in identifying which of those services would indeed be relevant and beneficial. Furthermore, evaluators need to develop a language that provides clarity and understanding among those who are not fluent in the professional jargon. 8

10 5. Basing the evaluation on comparative data and on tools that have already been successfully tested: It is the evaluators' responsibility to know their field well, to try to use instruments, methodologies and analytical methods that have been tested successfully in prior research, and to relate to comparative data that is accessible for them. These will allow them to evaluate programs from a broader, comparative perspective, and will assist in generalizing the conclusions from one program to other, similar, programs. 6. Promotion of inter-organizational cooperation: It is important to set up interorganizational platforms that will allow the sharing of knowledge regarding programs, including insights arising from evaluation research. Moreover, research has shown that thought is now being given to a transition from program evaluation to evaluation of a field as a whole. Such evaluation would examine all the interventions being carried out in a particular field, which have common goals or common target populations. The importance of such evaluation lies in its providing direction for those wholly active in the field, both in terms of their ability to influence and in terms of their success. It would appear that the social arena is ripe for this type of cooperation. In light of the findings and the discussion, we recommend that action be taken to implement the insights and ideas that this evaluation research has raised. For example, through publication of the evaluation report among broader audiences, holding workshops and seminars regarding the findings and the insights arising from them, building a website that will provide access to knowledge gained by evaluation research to all funding agencies. In addition establishment of a team that will propose a more up-to-date perception of the evaluator s role and ways to implement it, and promotion of the idea of evaluating a whole field through inter-organizational cooperation. Directions for further research are detailed in the full report. 9