Impact Evaluation Matters: Enhanced Learning Through Involving Stakeholders in Oxfam s Impact Studies evaluation Matters

Transcription

1 Peter Huisman, Rik Linssen, Anne Oudes Impact Evaluation Matters: Enhanced Learning Through Involving Stakeholders in Oxfam s Impact Studies evaluation Matters

2 The World Citizens Panel (WCP) has been developed by Oxfam as a new approach to impact measurement. Key characteristics of this approach are a combination of quantitative and qualitative research, involvement of all stakeholders throughout the process and a focus on joint learning. Oxfam has implemented the approach in six countries in Asia and Africa so far, while preparations in eight other countries are in progress. This article explains the approach and presents some of the outcomes from the impact studies recently done in Uganda and Nigeria. 75 Seeing the impact in the lives of people with my own eyes improved my way of working, Adrine shared at the last meeting of the World Citizens Panel in Uganda. I have become more motivated and determined to work towards seeing more and bigger changes in the lives of people. I know that our trainings can make a real difference, so I want to give my best to help achieve this impact. Adrine has participated in all steps of the World Citizens Panel impact measurement trajectory in Uganda, from the identification of impact indicators and survey questions to the data collection both questionnaires and in-depth interviews and interpretation of the outcomes. Her participation has made her see the impact of her work in a new way. She is one of the many stakeholders that have been involved in the implementation of the WCP. The World Citizens Panel approach The World Citizens Panel has been designed by Oxfam to measure the impact of its programs on people living in poverty and injustice. The WCP aims to measure in a simple and reliable manner the changes in the lives of people to which Oxfam and partners have contributed. It is a flexible approach that can be adapted to different needs and contexts. Our aim is to do this in all countries where Oxfam works and create a world citizens panel of people influencing Oxfam s strategies and activities. The approach is based on the following characteristics. Citizens central We aim to empower participants in our projects to speak about changes they have experienced and their perceptions on developments in their country. That is why we directly ask them about changes in their lives. We do not only want them to speak out, but also enable them to influence our activities. Based on their experiences and reflections, we critically review our work and impact achieved in order to render our work more effective. Focus on learning The primary goal of the WCP is to enable our program and partner organisations to learn and make decisions based on evidence. We involve our staff and staff of our partners in the design of the research (development of indicators), data collection and interpretation. People learn best when they are involved in the research process and the interpretation of its outcomes, as the quote from Adrine shows. They learn both how to measure impact as well as how they can increase their impact by making their strategies more effective. The WCP organises several moments to discuss the process and the outcomes with all stakeholders, to ensure that learning takes places and that lessons learned help adapt current strategies and shape new ones. Combining quantitative and qualitative research To measure the effectiveness of our work a survey is carried out among a random sample of our target group (project participants) and a comparison group (non participants), asking them about changes they have perceived in their lives. The aim is to measure the extent to which Oxfam and its local partners have contributed to these changes. The surveys make use of survey software that enables online as well as offline data collection using an A quarterly knowledge publication from Independent Development Evaluation at the African Development Bank Group

3 76 app on a smartphone or tablet. Descriptive statistics are available online immediately after finishing the data collection. Further statistical analyses are done by our WCP research team to determine the impact of the project activities. The surveys are complemented with qualitative research: people are asked to share their story of change, telling what has been the main change in their life/community in recent years and how this change has come about. The qualitative research complements the quantitative survey in a number of ways: where the survey provides insight in the what, the stories shed a light on the how, by providing details about how the change came about. The stories of change approach also leaves ample room to reveal unexpected changes. The findings can later be quantified, in a next round of surveys. Informed decision making based on reliable information Random sampling for the survey, careful selection of a comparison group, proper training and guidance of field staff to conduct the interviews, and robust statistical and qualitative analysis are key elements of the approach. Data collected by someone who is in regular contact with the respondent and has built rapport increases the reliability of the data, if done properly. External interviewers may be more neutral, but also less able to ask sensitive questions or encourage the respondent to be as open and frank as possible. Training and guidance is needed in order to avoid socially desirable answers (if the respondents want to please interviewers and therefore report more impact than is actually achieved). Also on these methodological issues, we want to make evidence based decisions. That is why our team of researchers is investigating the influence of interviewers from the organisation itself versus externally hired interviewers. Building local capacity Civil society organisations play an important role in fighting against poverty and injustice. They are an important countervailing power against structures of the state and the commercial institutions of the market. Oxfam therefore works with many civil society organisations in various countries. In all our programs, projects and campaigns we aim to strengthen the capacity of civil society. It is part of what we consider program quality. We believe that it is impossible to address poverty and injustice in a sustainable way, without strengthening the capacity of citizens to find solutions to their needs and influence decisions that affect their lives. Supporting the evaluation Matters

4 development of partners capacities in the area of planning, monitoring, evaluation and learning is one of the important components of our capacity building work with partners. The World Citizens Panel therefore starts with a workshop to explain the impact measurement methodology and provides training and guidance during data collection, analysis and reflection. Keeping costs at a minimum Impact studies usually require substantial investments of time and money in order to make sure that appropriate expertise, methods and systems are in place for reliable and meaningful data collection and analyses. Oxfam acknowledges the importance of high quality data collection and analyses in view of the strategic decision making that takes place based on the impact studies. At the same time, resources are limited and there is always a tension between allocating resources to direct project related activities aimed at changing the lives of people in poverty and injustice and measuring the impact of these activities. Therefore the World Citizens Panel developed a cost-effective yet reliable approach for data collection by working with local staff, mobile data collection, and relying to a large extent on automated systems. Implementation of the World Citizens Panel The WCP approach is being implemented gradually in programs within various countries. After a pilot in Uganda, impact studies have been carried out in Somalia, Nigeria, Mali, Pakistan and Cambodia. An example of a recent study in Nigeria is provided below. The World Citizens Panel approach was recently implemented in Nigeria. In a start-up workshop, we defined relevant impact indicators with Oxfam s partner organisations based on their theories of change. In developing these program specific indicators we aim to strike a balance between applicability of indicators to the relevant country and programmatic context and comparability with other existing indicators. Comparing the results of the WCP with other existing, nationally representative surveys (e.g. Demographic Health Surveys), can serve as a basis for triangulation. 77 Figure 1 Perceived Change in Gender Based Violence During the Past 5 Years in Nigeria Figure 2 Percentage of Respondents that Feels Able to Speak out Against VAWG in Their Community Looking back at the last 5 years, do you think there has been a change ind the occurrance of gender based violence in Nigeria? women Impact 1 5* men 14* 0% 25% 50% 75% 100% There is more gender based violence There is no change There is less gender based violence 1 The impact estimation is based on propensity score matching which is a statistical technique to ensure a fair comparison between the target and comparison group. The technique matches individuals in the target group to individuals with comparable characteristics (e.g. age, education, household size etc) in the comparison group. The matching is based on a set of relevant background characteristics (see the full report). The impact estimates here represent the difference between the target group and a matched comparison group (in the terminology of propensity score matching this is called the Average Treatment Effect on the Treated or ATT) on the relevant indicator. Target group Comparison group Source: WCP Nigeria, 2014, n Source: WCP Nigeria, 2014, partners in gender program n = 1341 *p<0.05 **p<0.01 A quarterly knowledge publication from Independent Development Evaluation at the African Development Bank Group

5 78 Representatives from partner organisations were trained in implementing the survey within their own organisations. The training in the workshops covered technical aspects such as how to select comparison groups, sampling, and how to train interviewers. Partner organisations then submitted survey implementation plans, covering technicalities, logistics, and budgets of their fieldwork. WCPstaff guided partner organisations through this process. Oxfam s partner organisations surveyed 1880 participants in their projects and a comparison group of 1427 respondents, who did not participate in these projects. The areas of focus of the programs in Nigeria and the questionnaire ranged from economic justice and livelihoods, to good governance, female leadership and gender justice. Violence against women and girls In Oxfam s rights based approach decreasing gender based violence is an important goal in the context of gender justice. Oxfam in Nigeria has identified a lack of space for women in social, economic and political areas as a major impediment to sustainable development. Unequal power relations between men and women become visible in the persistence of violence against women and girls in Nigeria. By creating awareness on the importance of this issue, women and girls are encouraged to speak out publicly and claim their space in the social, economic and political realm. After the data collection, the results of the survey were presented in a reflection workshop. During this reflection workshop partner organisations and Oxfam staff engaged in constructive and critical discussions on the impact of their work. The reflection workshop highlighted surprising findings, and defined domains for in-depth research using Stories of Change. During the reflection workshop we found that the impact survey shows that a large share of people perceive a decrease in gender based violence (GBV) during the past five years (see Figure 1). However, there seems to be disparity among respondents as there is also a large share (27 per cent) that argues that GBV increased in Nigeria. There seems to be increased awareness around gender based violence in both the target and comparison group based on various other indicators in the questionnaire. And Figure 3 Order of Action Taken Upon Experience of Gender Based Violence 100% 75% 50% 25% 0% 1st action 2nd action 3rd action 4th action Order of action Talked about it confidentially Has sought protection outside house Went to legal authorities Talked about it openly No 2nd/3rd/4th action Source: WCP Nigeria, 2014, selection of partners in gender program n = 742 evaluation Matters

6 beyond awareness, both men and women claim that they are able to speak out publicly against VAWG in their communities, as shown in Figure 2 (though to a lesser extent in the comparison group). The survey also inquired to what extent women had taken action upon experiencing gender based violence and asked women to rank the order of actions taken. This is displayed in Figure 3. We find that, despite the fact that respondents claim to be able to publicly speak about VAWG, very few women take actions that move beyond the private sphere upon experiencing GBV. Some women do, but most don t. This is the puzzle with which the qualitative stories of change process started. Which mechanism can explain this surprising finding from the survey? What factors can explain this mismatch between these attitudes and actually taking action upon experiencing GBV? What are the success stories behind the women who did take action? What were the blocking factors for women who did not move beyond discussing this in the private sphere? And, what can we learn from these stories to improve our theories of change? The stories will be analysed and combined with the results of the survey, so that project staff can reflect on the results of the impact study in a comprehensive manner and decide what adjustments in their strategy and project implementation will be needed. The capacity building and learning process has been valued by Oxfam s staff and partners in Nigeria. They have indicated that the capacity to measure the impact has been a key driver for engagement with the World Citizens Panel process. The reflections and learning has empowered them to make more critical decisions based on the understanding of what project participants truly consider change in their lives. Partners involvement in the WCP has made it possible for them in a relatively easy and affordable way, and in many cases for the first time, to learn what the impact of their work looks like. While the stories of change methodology is looked upon with enthusiasm because of its value for understanding the change processes and finding unexpected results, some partners have pointed out that the new capacity to understand the change their work is creating will also help them to tell about all the hard work they do in fighting injustice and poverty and support their fundraising efforts. For more information on the approach and results from the impact studies carried out so far, please visit 79 Profile of the Authors Peter Huisman is the project leader of the World Citizens Panel. He has worked at Oxfam Novib for almost 20 years, in different capacities. Prior to that he worked for the FAO in Ethiopia and Togo, among others. Peter has a background in human geography and a keen interest in innovation regarding monitoring, evaluation and learning. In 2012 he started the World Citizens Panel to strengthen Oxfam Novib s capacity to measure the impact of programs at participants level and to build the monitoring and evaluation skills of Oxfam Novib s partner organisations. Rik Linssen is a quantitative researcher specialising in survey research and econometric and statistical techniques. At Oxfam Novib, Rik works on quantitative impact measurement in several countries in Africa and Asia. Before joining Oxfam Novib in 2014, Rik worked as a statistical researcher at Statistics Netherlands. He obtained a research master s degree in social and cultural science at Radboud University Nijmegen and is currently finishing a PhD dissertation on trends in and determinants of political participation in a cross-national comparative perspective at Maastricht University. Anne Oudes is a qualitative researcher specialising in monitoring & evaluation and stories of change (Most Significant Change). She obtained a research master s degree in social sciences (cum laude) at the University of Amsterdam. At Oxfam Novib, Anne is responsible for qualitative research within the World Citizens Panel. Anne previously worked at the Ministry of Foreign Affairs and Simavi and also spent three years in Rwanda as a monitoring and evaluation expert for Belgian NGO Broederlijk Delen. A quarterly knowledge publication from Independent Development Evaluation at the African Development Bank Group