ANNEX I LRPS-2017-9130571 LOT 1 - WASH STBM ENDLINE EVALUATION SURVEY TERMS OF REFERENCE 1. Title of the assignment: WASH STBM endline evaluation survey. This survey is part of the UNICEF support to the Sanitasi Total Berbasis Masyarakat (STBM, or community approaches to total sanitation in English) support to Government of Indonesia, in partnership with UNICEF and the Bill and Melinda Gates Foundation (BMGF); the primary geographic focus of the supported work is in eastern Indonesia 2. Background and Justification: In partnership with the Government of Indonesia (GoI), sub-national Governments and the Bill and Melinda Gates Foundation (BMGF), UNICEF initiated a programme in Eastern Indonesia from 2013-2017 in 6 selected districts across 3 provinces in Eastern Indonesia, to further increase Government capacity and to accelerate scaling-up of the National Sanitasi Total Berbasis Masyarakat (STBM) programme across these provinces. UNICEF also has shared learning to National level for replication and acceleration of STBM and will share further on completion of the program. STBM is a national programme and it has been stated as a high priority for the President of Indonesia and the Ministry of Health to achieve universal access to sanitation by 2019. The programme originally aimed to increase the number of households using latrines and to achieve more than 40% of communities declaring Open Defecation Free (ODF) within the selected districts. Open defecation free status is normally granted by the District after a verification process by the District POKJA AMPL (WASH working group). This process follows the nationally-set ODF criteria by the Ministry of Health, available on the STBM website. A baseline and Knowledge Attitude and Practice (KAP) household survey was completed in February, 2014, aiming for establishing a baseline of sanitation pre-implementation status in the targeted communities. The survey collected data from 1700 households in the districts of Jayapura (Papua), Luwu Utara, Takalar and Barru (South Sulawesi), and Alor and Sumba Timur (NTT) and estimated access to sanitation in each of four sub-districts per district. Interim reports from field offices based on local monitoring by District Health Offices have shown increases in sanitation coverage and the number of ODF communities (for this program the dusun was normally regarded as community as triggering was done at this level) in focus districts during the course of the programme. Prior to the end of the BMGF programme in November, 2017, an evaluation will be conducted in areas already declared ODF and where field activities have been completed. This evaluation is aiming for evidence-based advocacy for effective programme and scale-up in order to accelerate efforts to achieve universal sanitation access in Indonesia; the national target is to achieve this by 2019. While key quantitative output indicators, including the number of latrines built in UNICEF-supported districts, percent of triggered communities achieving ODF, percent of households with access to sanitation that belong to the poorest two quintiles, number of programme/non-programme districts with plans and budget for implementing / scaling up STBM, will be evaluated through secondary data (i.e. Indonesian government data such as a national socio-economic survey, STBM SMS-based routine sanitation monitoring data reported by sanitarians, sub-national government planning documents, etc.), effectiveness and sustainability of the programme, which require more detailed data collection at community, and at household-level, cannot be assessed through secondary data analysis. Moreover, despite significant efforts to achieve ODF, the extent of and characteristics of sub-optimal household latrine use in ODF communities are poorly understood in Indonesia, resulting in unclear post-odf programmes. Therefore, an endline evaluation survey will be conducted with the focus on the effectiveness and sustainability of the joint Government-UNICEF STBM programme. Using a mixed qualitative and quantitative method, the survey will target UNICEF + Govt-intervened ODF communities in order to best capture (1) effectiveness by examining internal (i.e. quality of pre-triggering, triggering, post-triggering, follow-up monitoring and verification processes) and external factors (i.e. technical, financial, institutional, environmental / physical, and social, at the local level) explaining success to achieve ODF, and (2) sustainability by examining presence, extent and patterns of slippage (e.g. proportion of households returning to OD in surveyed ODF verified communities, and its disaggregation by wealth quintile), creation and strength of social norms associated with OD at community- (e.g. sanction / reward mechanisms to prevent OD) and household-level (e.g. belief around unacceptance of OD and a feeling of potential for social sanction within the community if seen defecating in the open), community challenges and measures against slippage, potential factors associated with sub-optimal latrine use (e.g. perceived latrine quality, water access for latrine use and maintenance, and size of households). This cross-sectional endline evaluation survey will also allow us to explore drivers and changes that led to ODF status through comparison between the baseline (pre-intervention) and the endline (post-intervention) by selecting a subset of surveyed ODF communities from the same sub-district where the baseline survey occurred (if and where possible). Changes may include strength of social norms against OD among households with sanitation access, and KAP (e.g. knowledge on link between OD PAGE 1
2 and diarrhoea, latrine use as a means of preventing child diarrhoea, safe child faeces disposal, handwashing behaviour etc.) However, a comparison analysis that would produce statistically significant indicators of change is not the primary focus of this study given the primary goal is around learning on effectiveness and sustainability and what worked for acceleration and scaleup on STBM in order to use this to push other Districts for replication. Undertaking a comparison of intervention and nonintervention villages or ODF/non-ODF villages is not proposed given the envelope of financial resources available. For instance, given the sample size of the baseline (n = 70-80 households per sub-district, four sub-district per district), more than two times larger household sample size per sub-district (n > 180 households per sub-district) will be required to detect 15% changes (25% to 10%) in strengthen of social norm (a two-sided test with α =0.05 and 80% power). Implication is that the total sample size would increase from the baseline of 1700 to required sample size of at least 4320. 3. Purpose of the assignment: The primary objectives of the endline survey in selected ODF communities of Jayapura (Papua), Luwu Utara, Takalar and Barru (South Sulawesi), and Alor and Sumba Timur (NTT) are: (1) At community level: to understand social norm dynamics in intervened communities (i.e. the process of creating & sustaining social norms via such parameters as shared factual beliefs in the community, social norms in these ODF communities, Empirical expectations, Normative expectations, presence of sanctions) via and key informant interviews and to estimate, via household surveys in selected ODF-verified communities, the slippage rates in these ODF communities. These two approaches will help to better understand the related challenges and community measures in creating and sustaining ODF status. (2) At household level: to assess key indicators around beliefs and actions (via KAP questions) and social norms as well as the key drivers and barriers to sanitation in ODF communities and its sustainability. Essentially, this will also enable us to examine differences from baseline household survey (in general communities) and endline household survey (in ODF communities), but actual comparison analysis is not the primary focus of this assignment as per the reasons outlined above. Asset questions will allow a wealth stratification to be undertaken in the analysis to disaggregate by socioeconomic status. The primary target audience of the evaluation include the UNICEF Country Office (CO), the Government of Indonesia, particularly relevant line ministries Ministry of Health (MoH), Bappenas (national planning agency), Ministry of Villages (MoV) and selected provincial and district line agencies, and the Gates Foundation Sanitation program for outcome reporting. The secondary audience may include other UN agencies and donor agencies working in the area of sanitation, hygiene and health/nutrition in Indonesia, NGOs, mass organizations, Private Sector agencies and development partners, particularly the World Bank. The findings will be presented to the Govt. of Indonesia and partners at a special learning session and will be used to push the key recommendations for replication and acceleration of STBM in Indonesia. The results will also be shared at Province level to report on progress and to also discuss on local replication and acceleration of STBM in that area. 4. Scope of Work & Methodologies 4.1. Locations Sampling will be taken place in six districts, namely, Jayapura (Papua), Luwu Utara, Takalar and Barru (South Sulawesi), and Alor and Sumba Timur (NTT). 4.2. Survey Design UNICEF sanitation programme, using the STBM/Community Approaches to Total Sanitation (CATS), aims to eliminate open defecation by closely working with communities to change their beliefs and expectations around sanitation. Furthermore, to achieve longer-term sustainability of sanitation interventions, it is critical to create and sustain new social norms and expectations in communities around their values and norms dealing with sanitation and community health. However, little is known about development of the social norms through sanitation intervention programmes and its association with sustainability of latrine use behavior. Moreover, despite substantial efforts to achieve ODF communities, a number of studies in other countries have suggested challenges to sustain ODF status and various slippage patterns have been seen in communities after achieving ODF. A UNICEF rapid survey conducted in three UNICEF-programme districts (Jayapura, Luwu Utara and Sumba Timur) in August, 2016 also reported that 20% of surveyed households, from a combination of both ODF and non-odf communities, did not show any sign of latrine usage at the time of survey, indicating that it cannot be assumed that access to a latrine guarantees consistent use of the latrine by household members. In some cases, there was a discrepancy between reported use at individual level and observed use at household level. It is, therefore, imperative that latrine use reported and actual behavior needs to be critically PAGE 2
3 evaluated in Indonesia. Baseline (Feb 2014) and the rapid survey (Aug. 2016) questionnaires and results will be made available to help to formulate the endline design and inform the inception report. This survey design is cross-sectional using a mixed quantitative and qualitative methods. Four ODF communities per district, total of 24 ODF communities (i.e. dusun), will be sampled. Selection criteria of ODF communities will be a community that UNICEF intervened with ODF verification at village-level (i.e. stop open defecation) at least one year ago but not earlier than 3 years ago (i.e. so can be linked to this program). Additionally, several ODF communities will be sampled in sub-districts where the baseline survey was conducted if and where possible, so that comparison between the baseline and endline might be made possible depending on the effect size. The survey will consist of (1) three per community, including male, female and those who may be difficult to sustain latrine use behavior such as poorer households, older people, farmers and newly migrated households, (2) five key informant interviews (Head of village, sanitarians, NGO partners, district health staff, Bappeda, etc.), (3) a household survey for all communities with sampling of about 90 households per community, allowing to estimate a 15±5% community-level slippage rate with 95% confidence interval, assuming a community size is 150 households (further adjustment may be needed due to different size of ODF communities selected), and (4) transect walk for all communities to observe presence of human feces in communities. Due to lack of slippage rate data in our programme districts, 15% slippage rate was chosen based on findings of the rapid survey. Given 20% sub-optimal latrine use in a combination of both ODF and non-odf communities in Luwu Utara, Sumba Timur and Jayapura, we estimated slightly lower slippage rate (i.e. 15%) for ODF verified communities. aim for capturing the processes to achieve ODF and any community measures such as community monitoring, social or financial sanctions, internal pressure mechanisms and toilet maintenance support systems to sustain ODF status at communitylevel, while a household survey will provide insights around household- and individual respondent-level latrine use behavior through visual observation of signs of latrine use and self-reporting (i.e. frequency of latrine use over time and space). A transect walk will allow to assess if open defecation areas are present as well as to see the general cleanliness of the community. Table 1 summarizes the key proposed approaches and tools. Baseline and KAP study questionnaire, which UNICEF will provide to a contractor in Bahasa Indonesia, will be used with some modifications, as and where needed, in order to estimate household sanitation access and other key indicators such as social norms and sustainability related issues. UNICEF/UNICEF partners will prepare a list of ODF communities and will inform the contractor, accordingly. The contractor will randomly choose ODF communities that meet selection criteria. The contractor will inform UNICEF and its partners of their schedule prior to surveys, while UNICEF and partners will be responsible for communication with government officials and other relevant persons. Table 1. Type of data and methods proposed for this endline survey Type of data to be collected Level of community compliance with original ODF criteria Quality of STBM preparation (before triggering activities - i.e. community meeting, prep for socialization, identifying community leaders, STBM implementation plan - location, timing, platform etc.) Quality of STBM process (triggering, attendance rates in triggering, development of time-bound community action plan, monitoring, post-triggering/consultation, selfdeclared (not verified), verification and declaration) Post-ODF activities (monitoring, re-triggering, activities/support for moving up the sanitation ladder etc.) Defecation practices in the community (i.e. group of people likely still practicing OD) and beliefs/taboos associated with sub-optimal latrine use esp. via a gender lens Community-level proxy indicators for measuring community commitment to sustain ODF status (i.e. sanctions, internal pressure mechanisms etc.) Drivers / barriers for changes to achieve ODF status & social norm dynamics Potential impact of Tinju Tinja campaign and other communication / advocacy approaches and tools Defecation practices in the household (i.e. consistent latrine use behavior at household- and individual-level via observation and self-reporting, child feces disposal) Methodology ODF check as part of HH survey & transect walk at short notice Key informant interviews Key informant interviews PAGE 3
4 Strength of social norms Potential factors associated with consistent latrine use behavior (i.e. satisfaction with facility, quality of a latrine, access to water etc.) and KAP factors at HH level. Limitations Number of sampled households per community (dusun) is calculated to estimate community-level slippage rate of 15% with 5% precision and 95% confidence interval, while number of ODF communities to be sampled (n = 24) is calculated based on available budge for the endline survey. Therefore, slippage rate estimated in this survey will not be district-level representative. This crosssectional study will be conducted over one month during dry season and will be in line with agreed reporting timeline as well as the need to respect the Ramadhan/Idul-Fitri period. Hence, potentially important temporal and seasonal variability of slippage patterns will not be captured. Further methodological guidance including sample size will be discussed and finalized at the inception phase. For Ethics approval, please refer to the documents that can be downloaded, listed below: Ethical guidelines for evaluations: http://www.unevaluation.org/document/download/548 Evaluators' code of conduct: http://www.unicef.org/evaluation/files/evaluation_principles_uneg_code_of_conduct.pdf Procedure for Ethical Standards: https://unicef.sharepoint.com/teams/oor/siteassets/sitepages/procedures/ UNICEFPROCEDUREONETHICSINEVIDENCEGENERATION.PDF Ethical research involving children: http://childethics.com/wp-content/uploads/2013/10/eric-compendium-approved-digital-web.pdf STBM: http://stbm-indonesia.org/?page=tentang-stbm&command=stbm&id1=2 Tasks Deliverable Timeframe/duration Inception report finalized after UNICEF Early May 2017 inputs incorporated; an important 30% of project payment element of this will be a clear methodology and shared understanding + agreement on how the social norms measurement will be undertaken Inception report submission, including implementation plan, objectives, methodology, time frame of the study, detailed sampling frameworks, data collection instruments (EN and Indonesian), application for ethical clearance and tabulation data plans Local Ethical clearance applied for and granted Data collection Headline results and raw data submission Final report submission with presentation to national stakeholders and key recommendations highlighted; UNICEF will arrange the venue Ethical clearance shared to UNICEF Draft report in English and Bahasa Indonesia containing information and analysis Power point Presentation to be shared during a meeting with UNICEF in English and Bahasa Indonesia. Power point slides summarizing headline results Cleaned Raw data files (in agreed format) Final report in English and final PowerPoint presentation in both English and Bahasa Indonesia: In addition, a separate 4 page summary evaluation brief should be provided. The format / outline of the report will be based on Early July (prior to data collection) Middle of July early August, 2017 (timeline is fixed due to reporting requirements and respecting the period of Ramadhan) 30% of project payment Late August, 2017 Middle of September, 2017 40% of project payment PAGE 4
5 5. Reporting Requirements: UNICEF s reporting standard with all related data and all tabulations (in agreed format) Inception report outlining survey methodology and work-plan with milestones Draft final report on both quantitative and qualitative aspects with descriptive statistics Soft copy of Raw data in Bahasa and English in Excel and SPSS format (depending on feasibility after questionnaires have been designed) PPT in English and Bahasa Indonesia and final report in English (no hard copies needed) 6. Timing/duration of contract: 6 months. The institution needs to provide an estimated and realistic timeline for each task/deliverables. 7. Qualifications Required: A consultancy/research agency with at least 3 to 5 references on data collection and quantitative survey over the last 5 years related to WASH / Health / Social and Behaviour change programming, especially in Indonesia. Experience in qualitative data collection (including Focus Group discussions) at local level in Indonesian villages Exposure to the concepts of social norms and their measurement Excellent verbal and written English for project managers and Bahasa Indonesia for local enumerators and field report writers Computer skills for data entry and analysis A team consisting of a project manager, field coordinator, a minimum number of enumerators and local data analysis experts so that the identified team should be able to respect the project timeline. PAGE 5