Oxfam PNG - WASH Program Final Evaluation

Size: px

Start display at page:

Download "Oxfam PNG - WASH Program Final Evaluation"

Cordelia Daniel
5 years ago
Views:

1 Oxfam PNG - WASH Program Final Evaluation Terms of reference Final Evaluation Program/project title /affiliate identification code Partner organisations Geographical coverage: Program Duration Oxfam PNG WASH Program 1. Code: SDF Round 2 (MFAT) 2. Code: PNG SEPIK WASH (DFAT ANCP) Highlands: Eastern Highlands Provincial health Authority (EHPHA) Appropriate Technology Projects Inc. of Goroka (ATP) Touching the untouchables (TTU) Kunaiku, Gerekane and Wandekane (KGWAN) Bougainville: Bougainville health, Sepik: Yagoru Sausia district Eastern Highlands, Simbu, Bougainville & East Sepik Provinces, Papua New Guinea 1. June, 2012 July 2016 (MFAT programme) 2. July 2013 June 2017 (DFAT Programme) Evaluation Period 30 days, May June 2016 Evaluation commissioning manager John Nokue Oxfam PNG WASH Program Manager 1. Background. This Terms of Reference (TOR) is for the final evaluation of two Water, Sanitation and Hygiene (WASH) Programs in Papua New Guinea (PNG): 1. The PNG WASH Program located in the Highlands and Bougainville Provinces and funded by the New Zealand Ministry of Foreign Affairs and Trade (MFAT), a four year program which ends in July 2016 and 2. The East Sepik WASH Program funded by the Australian Department of Foreign Affairs and Trade (DFAT), a two year program which ends in June The Highlands and Bougainville WASH Program s aim is to improve the health and resilience of vulnerable communities through the provision of sustainable safe water supply, improved sanitation facilities and hygiene practice in PNG. The WASH program for Bougainville and Highlands has 1 long term outcome, 1 medium term outcome and 3 short term outcomes: Long Term Outcome 1: Medium Term Outcome 1: Improved hygiene practices and improved access to safe water supply in target rural communities, schools and health facilities Rural communities, schools and health facilities have improved hygiene practices and improved access to clean water supply

2 Short Term Outcome 1: Short Term Outcome 2: Short Term Outcome 3: Community led construction of water supply and sanitation infrastructure Rural communities, schools and health facilities improved hygiene practices, and have improved awareness of natural and local hazards, including disease outbreak. Local administration, partner organisations and communities have improved skills and understanding in how to work together in WASH sector activities The Sepik WASH program s aim is to improve public health by increasing poor women and men s access to safe water and sanitation in East Sepik. The outcomes of the project are: 1. Increased access to sanitation, hygiene and clean water; 2. Reduced gender inequality relating to access to and management of sanitation, hygiene and clean water; and 3. Strengthened Water, Sanitation and Hygiene (WASH) capacities of target communities so villagers could maintain basic water systems and hygiene practices. Delivery Model of Program: The WASH program is currently operating out of 3 geographic locations with a WASH coordinator leading implementation in each location, and a MEL officer and WASH program Manager sitting across all three programmes. In the Highlands and Bougainville, the program is exclusively delivered through partners while the Sepik Program is delivered through a mixture of partnerships and direct implementation by Oxfam. The Oxfam WASH Program has partnered with government sectors such as district and provincial health departments and key community based organisations to implement WASH activities where they have a presence and are already working in target communities, schools and health facilities. The partner organisations have supported Oxfam to implement WASH activities in both software and hardware. Between each partner and Oxfam there is an MOU which outlines the responsibilities of each organisation in the delivery of different types of WASH system construction or/and health and hygiene education training. 2. Purpose and Audience of the evaluation The purpose of the evaluation is twofold, it is about accountability, upwards accountability to the donors, DFAT and MFAT who have funded the two programs and importantly downward and social accountability to the partners and communities who Oxfam has worked with and where the programs have been implemented. The external evaluation will be useful for MFAT and DFAT to understand the impact of the program and the effectiveness of the approaches used to improve the lives of rural people in PNG. For partners, communities and government the evaluation findings and recommendations can be used to understand what has been the impact of programs and what is best practice which can be continued through other funding or replicated in other communities. The purpose of the evaluation is also about learning and future program improvement, particularly for the WASH program team and Oxfam in PNG. The evaluation of the programs and essentially two models (Bougainville/Highlands and Sepik) presents an important opportunity to reflect on and learn from Oxfam PNG s approach to WASH, both in terms of the

3 delivery model and ways of partnering and influencing stakeholders. The evaluation will provide evidence and judgements on the strengths and weaknesses of the two programs in order to inform future decisions about Oxfam s role and value add in the WASH sector in PNG. It should inform what programmatic and influencing role Oxfam should play moving forward and provide clear guidance about how Oxfam can improve its engagement with key stakeholders (private sector, government and community organisations). The final evaluation report will be shared by Oxfam with the New Zealand Government (MFAT) Australian Government (DFAT), local partners, project beneficiaries and other stakeholders within the WASH sector circle 3. Evaluation Objectives Objective 1: Impact and Sustainability The evaluation will assess the significant changes experienced in the lives of women, men, girls and boys as a result of the WASH work with partners in targeted schools, health facilities and communities. The evaluation should specifically consider the impacts (both intended and unintended) of the program on women and vulnerable groups including people living with disabilities. In addition the evaluation will assess whether the changes realised are likely to be sustained over the long term and past project closure, and assess the strategies used by Oxfam and partners to promote sustainability. Objective 2: Effectiveness The evaluation will articulate and test the Theory of Change for the two programs and assess the reasons behind the achievement or not of the objectives, and whether the gaps lie in the TOC or with difficulties in implementation. Both the strategies and outcomes of the hardware and software components need to be assessed for effectiveness. Objective 3: Relevance and Appropriateness The evaluation will assess the appropriateness and relevance of the current delivery strategies and model of partnering in relation to the context and also the role and value add of Oxfam in this space. It should also look at how effectively and appropriately we have worked with partners and built their capacity. Objective 3: Value for Money The evaluation will look at to what extent the program represented the best possible use of available resources to achieve outcomes of the greatest possible value to communities. 4. Evaluation Questions The following are key overarching questions to guide the evaluation based on the prioritized evaluation criteria above of impact and sustainability, effectiveness, relevance and value for money. As part of the evaluation, following a desk review of the relevant documents, the evaluator is expected to work with the evaluation management committee and the evaluation team to develop in a participatory way the final version of the key evaluation questions, the sub questions, and correspondingly the methods which will be used.

4 Criteria Impact and Sustainability Key Evaluation Questions What significant changes (positive, negative and unintended) have occurred for women, men, girls and boys and marginalized groups in the program target communities and facilities (schools/health facilities)? Are there differences between the groups? In what ways has the program been successful in improving gender inequality and empowering women in the target communities? What indications do we have that the changes the programs contributed to will last? Effectiveness To what extent have the strategies used led to changes in access, practices and policy? What assumptions did we make in choosing the different strategies? Did these assumptions hold true regarding how change will happen? Are there alternative strategies which could have been more effective? What processes have enabled or hindered the achievement of outcomes? Relevance What is the appropriateness of the delivery model and the contribution or value add of Oxfam in the WASH sector in PNG? How relevant have the partnerships been between Oxfam, community organizations and government sectors, and what difference have the partnerships made to the program outcomes? How effective and relevant a partner has Oxfam been? Value for Money Were the choices about allocation of resources the most efficient and effective? To what extent has the distribution of resources to specific activities/strategies in the programs been appropriate? Was the program implemented in the most efficient way compared to alternatives? 5. Scope of the evaluation, approach and methods to be used The evaluation should take a utilisation approach to ensure that the findings and recommendations are actionable and relevant to the program team and to Oxfam in PNG. The evaluation should also be considered as an evaluation capacity building opportunity for the Oxfam in PNG WASH team. The evaluation plan and methodology should be as participatory as possible in relation to involving the WASH team in the design of the methodology and the data collection and analysis tools, the actual data collection, and contributing to sense making of the data. Where possible, the involvement of partners or other stakeholders should also be considered in the evaluation design and process. The scope of the evaluation needs to be determined by the evaluator with the evaluation management committee (which will be formed) to ensure a representative, but realistic sample is chosen for the field data collection. The evaluator will be provided with project

5 documents and mapping of partner activities, stakeholders and communities to determine the appropriate sample size. The evaluation should undertake a mixed methods approach and data collection and analysis methods should be selected based on their appropriateness to answer the key evaluation questions and based on what is realistic in the timeframe and the resources available. The evaluation methods should be decided upon in a participatory fashion with the evaluation management committee. The evaluation is expected to be gender responsive and explore issues of gender equality through the questions chosen and methods used. The evaluator is expected to produce an evaluation plan and methodology as a first deliverable before the field work commences. Training of the evaluation team in the methods is also an expected role of the evaluator. The evaluation should adhere to Oxfam Australia s ethics guidelines. The consultant will be provided with a copy of the Ethics Guidelines and Oxfam Cultural Protocols and will be expected to actively ensure that they are followed. 6. Evaluation Team The Oxfam in PNG WASH manager will be the commissioning evaluation manager, there will also be a WASH evaluation management committee comprised of relevant staff from Oxfam PNG, Oxfam Australia and Oxfam New Zealand to provide guidance and technical support throughout the evaluation process. The consultant/s is expected to liaise with the evaluation manager and evaluation management committee to ensure input and guidance to the different phases of the evaluation (please see phases below). 7. Schedule, budget, logistics and deliverables. The daily rate for the evaluation is $ per day with an expected timeframe of 30 days over the period May-June The distribution of days for certain phases will be finalized once the sample size and locations for field visits is confirmed and the methodology is developed. The evaluator can also suggest changes to the schedule based on expected timeframes for each phase and deliverable. Travel costs associated with the consultant will be covered by Oxfam. Travel costs and additional expenses for participants in order to facilitate face-to-face data collection will also be covered by Oxfam. The logistics for the evaluation will be arranged by Oxfam and the Oxfam partner organisations. The Oxfam partner organisation will support mobilising communities and institutions prior to visits and interviews.

6 Activity Phase 1 Inception Scoping and Preparation Desk Review of program documents Planning with Oxfam Staff Phase 2 Participatory Evaluation Plan and Methodological development Designing methodology/sample/pilo ting tools with evaluation team, and partners (if possible) Training of evaluation team No. Of Consultant Location days 2 days Home base of consultant 3 days Home base/in country - TBC Deliverable Evaluation Plan (with methodology) Phase 3 Data Collection Data collection in the field Phase 4 Data Analysis and Interpretation, Participatory Sense Making Data Collation and Analysis Data sense making and verification workshop with program team, country team and partners (if possible) 16 days (including travel days) Exact timeframe to depend on field locations and logistics In country Emerging Data reports 3 days In country Presentation of preliminary findings Documentatio n of the sensemaking/validat ion workshop Phase 5 Data Synthesis Draft Report Feedback from evaluation team, evaluation management committee, partners Final Report with incorporated feedback 5 days (to be split between drafting and then finalizing post feedback) Home base of Consultant Draft Report Final Report 8. Evaluation Report The final evaluation report needs to consider the different users and audiences for

7 accountability and learning. Proposals of modified reports and creative formats for different users and audiences are encouraged. The report format/s will be clarified during the evaluation planning stage. The evaluation consultant will be provided with Oxfam report writing requirements, including branding, and the final report will be the property of Oxfam. Oxfam owns all data and resources generated by the evaluation, including participant verbal and written feedback, documentation and reports. The consultant is required to document and provide all raw and analyzed data to Oxfam. A comprehensive version of the evaluation will be published on Oxfam International s internal website as well as on Oxfam Australia s and Oxfam New Zealand s external website. 9. Consultant qualifications and experience The consultant may present as an individual or as a team, the consultant or team should have the following qualifications and experience: Qualifications and/or demonstrated experience working in and leading evaluations for rural development and water, sanitation and hygiene programs, preferably in the Pacific or PNG specifically. Demonstrated ability and experience in evaluating international development programs using quantitative and qualitative methods and participatory approaches. Knowledge and experience of evaluation capacity building and training local teams in data collection and analysis methods. Knowledge and practical experience in the application of gender conceptual frameworks and analysis during evaluation processes. Excellent oral communication skills and the ability to deliver high quality written reports in English. Ability to communicate well in Tok Pisin. A Bachelor s degree in a relevant field, postgraduate qualifications will be highly regarded. Strong interpersonal and facilitation skills, and cultural and gender sensitivity in working with local communities and a range of partners and stakeholders Good working experience in PNG with understanding of the PNG context. In line with Oxfam s 2020 vision we welcome applications from Pacific Islanders who meet the criteria as outlined above. 10. Key terms and conditions It is anticipated the consultant will be paid in three tranches, as follows: Tranche / Milestone completed Percentage / amount Date of transfer (TBD upon recruitment)

8 1) Submission of evaluation plan with methodology 20% 2) Submission of Draft Report 55% 3) Submission of Final Report 25% Payment will be paid upon issuing of Invoice on delivery of deliverables achieved within timeframe. Oxfam PNG will request evidence of insurance cover prior to commencement of the assignment All equipment and materials required for the assignment are to be provided by the contractor except where otherwise indicated in the Terms of Reference 11. Submission procedure Consultants interested in applying for the evaluation consultancy are required to submit the following: 1. A proposal addressing the required qualifications and experience, a brief proposed approach and suggested methodology based on the information in the TOR (if proposing a team approach, please specify roles and responsibilities), confirmation of daily rate, and indication of availability to undertake the evaluation within the preferred timing. 2. Recent CV with names and contacts of three referees 3. An example of a relevant piece of written work (preferably an evaluation report). Closing date for Applications: 18 th April 2016 Any questions and clarifications about the Evaluation TOR and process should be ed to the following: John Nokue, WASH Program Manager, Oxfam PNG johnn@oxfampng.org.pg (PNG) and Kathryn Robb, Pacific Program Quality Coordinator, Oxfam Australia kathrynr@oxfam.org.au (Australia) Proposals should be submitted via to johnn@oxfampng.org.pg and kathrynr@oxfam.org.au

Activity Design for Oxfam in Papua New Guinea WASH Project and WASH Manual

Activity Design for Oxfam in Papua New Guinea WASH Project and WASH Manual DATE LOCATION TOTAL NUMBER OF DAYS REPORTING TO: To be completed by end of April 2018 or upon negotiation with selection panel