Gender in Project and Program. Padma Karunaratne

Transcription

1 World Bank Group Post Conflict Reconstruction: Regional Capacity Building Workshop in Engendering Economic Policy, Gender in Statistics, and Gender in Monitoring & Evaluation Gender in Project and Program Monitoring & Evaluation by Padma Karunaratne Amman, Jordan May 18, 2005

2 Purpose of the Session Define gender Present the steps in the monitoring and evaluation process Show how gender can be integrated into each step in the M&E process 2

3 Session Objectives To make the participants understand: basic research concepts and techniques that can be used in monitoring and evaluation ways in which gender can be integrated into each step in the monitoring and evaluation process 3

4 Defining Gender Culturally based expectations about roles and behavior of men and women. Sex is biologically determined; Gender is socially constructed. Gender roles vary among societies and change within any given society. Gender-related beliefs are deeply embedded and there may be strong resistance to change. 4

5 Defining Gender Gender is a sociocultural variable that refers to the comparative, relational, or differential roles, responsibilities, and activities of females and males.. Gender roles vary among societies and change within any given society. Gender roles could vary over time. 5

6 Defining Gender (Continued) Gender does not mean women only In developing a project, it is important to understand the gender roles and design projects within that context For example, health programs to reduce fertility and maternal and infant mortality: How do the roles and relationships of men and women impact family planning and maternal and child health programs? How do gender roles and relationships impact the potential for success of the project? 6

7 Monitoring & Evaluation (M&E) Monitoring is a continuous internal management activity to ensure that the program implementation and on-going operation are on track. Evaluation is an internal or external management activity to assess whether a program is achieving its intended objectives. 7

8 Importance of Gender in M&E Men and women have different development priorities, needs and constraints. Men and Women are affected differently by development programs. Standard planning methods often do not capture these differences. 8

9 Importance of Gender in M&E (Continued) Efficiency/ equity enhanced when gender differences taken into account. Conventional M&E systems often do not capture gender differences in access and impacts. 9

10 Issues and Challenges in Gender Sensitive M&E Assumed gender neutrality of M&E methods and processes M&E units directed and composed by technical staff who lack awareness of gender issues. Women are often under-represented in evaluation and interview teams. 10

11 Issues and Challenges in Gender Sensitive M&E (Continued) Women cannot express themselves freely because many data collection methods make it difficult they may be less fluent than men in the language used in the interview they may not be allowed to represent or speak in village meetings and community consultations. Obtaining information from both women and men may increase the cost and time of data collection. 11

12 Steps in M&E Process Planning Doing Reporting 12

13 Steps in M & E Process (cont d) Planning Determining Your Questions Selecting a Monitoring and Evaluation Design Strategy Identifying Your Measures and Measurement Strategy Developing Your Data Collection Strategy Identifying Your Data Analysis Strategy Reviewing and Testing Your Plan 13

14 Steps in M & E Process (cont d) Doing Gathering the Data Preparing Data for Analysis Analyzing and Interpreting the Data 14

15 Steps in M & E Process (cont d) Reporting the Results Executive Summary Use of Charts and Tables Oral Briefings 15

16 Design Matrix What is Design Matrix and why use it? 16

17 Design Matrix A tool that can help focus on all the details It is a visual tool Focus is on content not writing style It is a living document Planning is an iterative process. This is a generic format Change it to fit your style 17

18 Program Design Matrix Questions Sub-Questions Type of Question Type of Design Indicators Measures Data Sources Sample 18

19 Program Design Matrix Questions Sub-Questions Data Collection Data Analysis Comments 19

20 World Bank Group Models, Assumptions and Causality

21 M&E Process Phase 1. Specify model and select design Define the problem/situation Specify the project (program) model Determine the questions Select the design 21

22 Model: Definition A simplified abstraction of a program Shows the major parts, or elements, of the program and their relationships Often depicted in a schematic overview 22

23 Models: Purpose Models help us.. Visualize a program in context Clarify program objectives Clarify program inputs Identify relationships among program components Think more analytically about cause and effect Identify key assumptions behind the program 23

24 What is your program? For example If you are launching a project or program in the health sector you will identify what your goals, objectives, and what you want to achieve during and after the project or program period. 24

25 What Difference Did It Make? You want to know the difference your intervention made For health service supply? For household behaviors and risk factors? For HNP Millennium Development Goals? 25

26 What Difference Did It Make? did more mothers and their children receive health care? did more women receive prenatal care? are there more trained health care workers providing services? did contraceptive prevalence increase? did it improve access to emergency obstetric care? 26

27 Example: Family Health Project Did the program increase the availability of services to families? What else might have affected the availability of services to families? Did the program decrease the fertility rate? What else might have affected the fertility rate? 27

28 What Difference Did It Make? How would we know? How would we determine whether key factors of interest, such as the HNP MDG indicators, would have changed without the project? This is the role of M&E 28

29 Models Help Understand Causality Did the program cause a change? Are the changes we observe a consequence of the program? What else besides the program contributed to the observed changes? Often depicted in a schematic overview How does gender influence on program outcome and impact 29

30 Generic Program Model Inputs Impacts Outcomes Outputs Activities 30

31 Gender in Program Models Are there differences in inputs and activities based on gender? Are there different outputs based on gender? Does the program cause different outcomes based on gender? Are there differences in impact on gender groups (causality)? 31

32 Case Discussion: Family Health or Microcredit What are major program inputs? Gender? What are the program s components? Gender? What are some program outputs? Gender? What are some program outcomes? Gender? What are some long-range impacts of the program? Gender? 32

33 Family Health Model Impacts Reduced fertility rate Outcomes Outputs Activities Inputs Number of participants who use birth control consistently Number of participants in education and health services Community Education Health services Money, Staff 33

34 Assumptions of Program Logic Models Logic models articulate program assumptions about Causal relationships internal to the program. For example, will the specified training result in improved service delivery? Which factors in the social and physical environment external to the program might influence program outcomes. For example, will a reduction in farm incomes affect the level of participation in the program? 34

35 Program Logic Model Components: Input: Resources people, dollars, books, facilities, etc. Activity: What the program does train people, provide services Output: Products or Services a calculation of effort or activity 35

36 Program Logic Model (Cont d) Components (continued): Outcome: Benefits to participants results, changes more immediate Impact: Long Term Benefits achievement of the program s/project s goals 36

37 Application to Child Mortality Program Logic Model: Inputs Activities Outputs Outcomes Impact Resources Services Products Benefits Changes money staff supplies Pre-natal Preventive Birth attendance Treatment for ARI More use of prenatal services More children immunized Healthy Children Healthy mothers Reduced child mortality Reduced poverty Immunization 37

38 Causality Causality is at the core of a logic model but it is not easy to establish. Necessary conditions: 1. Technical common sense 2. Time order 3. Co-variation 4. Elimination of rival explanations Remember: association does not equal causation! 38

39 Causality: necessary conditions Technical Common Sense Time order The relationships must make sense, from a technical (practical) perspective. The cause must occur before or at the same time as the effect Covariation As the independent variable changes, so does the dependent variable. This change is, ideally in the direction you have predicted. Elimination of rival explanations You must eliminate all other possible rival explanations, i.e., show that there is no other plausible reason why the change occurred. 39

40 Summary: Models, Assumptions and Causality Logic models clarify program objectives, causality and assumptions Assumptions should be made explicit in planning, and noted in monitoring and evaluation Causal relationships need to be examined carefully 40

41 Exercise: 1 Briefly describe a project/program that is the focus of your work (or will be the focus of your work). Identify what the project/program intended to accomplish and the activities used to accomplish goals. Why would the project/program be a good candidate for Monitoring or/and Evaluation? What are some challenges for monitoring/evaluating the project/program? What are some gender issues? 41

42 Monitoring & Evaluation (M&E) Monitoring is a continuous internal management activity to ensure that the program implementation and on-going operation are on track. Evaluation is an internal or external management activity to assess whether a program is achieving its intended objectives. 42

43 Monitoring (cont d) Program monitoring is the periodic measurement of progress of a program. Monitoring may focus on: Comparison of performance with existing program objectives; or Assessment of intended and unintended outcomes of a program 43

44 Evaluation (cont d) A systematic search for answers to questions about: Design Implementation Effectiveness 44

45 Types of Evaluations Process: operations and service delivery are there problems in service delivery? Cost-effectiveness: costs and results Impact: Did the program make a difference? Immediate or long term? 45

46 Purpose of Evaluations Formative evaluation Provide interim feedback on performance progress. Assessment occurs periodically during implementation. Performance monitoring (collecting program performance data and incorporating the analysis into program management) is a type of formative evaluation. Summative evaluation Assess program outcome/impact. Assessment occurs at the end of the program. 46

47 World Bank Group Evaluation Questions

48 Learning Objectives To understand the different types of evaluation questions To apply this understanding to your projects/programs 48

49 Steps in M&E Process Planning Determining Your Questions Selecting an Evaluation Design Strategy Identifying Your Measures and Measurement Strategy Developing Your Data Collection Strategy Identifying Your Analysis Strategy Reviewing and Testing Your Plan 49

50 Determining Your Questions What s the purpose of your study? Consider the goals and objectives of the program, in light of sectoral and macro-level objectives Consider the stakeholders Decide on your evaluation or monitoring questions Use models to help identify and clarify questions 50

51 Defining Your Question (cont d) What is it you want to know? Who wants to know? Why do they want to know? How will the results be used? What s most important? Operationalizing your questions (move from general to specific) 51

52 Sources of Questions Project/Program documents Project, Government, NGO officials Program Beneficiaries Experts Other studies 52

53 Microcredit or Family Health (Case study) Discuss: What are the components of this project/program? Who is likely to be interested in this project/program? What questions are they likely to have? What questions would you want to ask about this program? 53

54 Types of Questions 1. Descriptive Questions: What Is 2. Normative questions: What should be 3. Impact or Cause/Effect Questions: What difference does it make? Do program participants have new skills? Do they have better paying jobs? Is the poverty rate reduced? Does this imply causality? 54

55 Descriptive Questions Seeks to understand or describe a program or process. Provides a snapshot of what is Journalist s questions: who, what, where, when, how, and how much? 55

56 Descriptive Questions (Cont d) Who receives the program? Gender? What are the characteristics of the program? Where is the program delivered? When was the program implemented? How do the participants feel about the program? How much did the program cost? 56

57 Descriptive Questions (Cont d) Can be used to describe: inputs, activities, outputs 57

58 Normative Questions Examine what is And compares it to what should be Are we doing what we are supposed to be doing? Are we hitting our target? 58

59 Impact or Cause-Effect Questions Impact or Causal questions: Examine the effect of a program. A Program Outcome Model: Identifies the outcomes and impacts of a particular program. Has the program made a difference? Is the change a result of the program? 59

60 Outcomes and Impacts Difference in terms of time Outcomes: more immediate changes Impacts: longer term changes Goal achievement Final results (expected and unexpected) 60

61 Questions: Summary Defining your evaluation and monitoring questions is a key step Be clear about types of questions (descriptive, normative or impact) 61

62 Introduce Gender in Question Development Critical Point If gender sensitive questions are not introduced here, you lose the opportunity to consider gender in later stages of the evaluation or in monitoring the program. 62

63 Exercise: 2 Based on your project/program develop one (monitoring or evaluation) main question and any number of sub-questions to answer that one question. Fill the appropriate column in the design matrix. 63

64 World Bank Group Monitoring & Evaluation Design Strategies

65 Developing Monitoring & Evaluation Design Strategy Definition: The overall strategy for how you will systematically gather and analyze data to answer your evaluation question. Design may not be an issue for descriptive and normative questions, but it is critical for impact questions 65

66 Developing Monitoring & Evaluation Design Strategy Design=Strategy for answering M&E questions: Did the program cause the changes you observe? 66

67 The Challenge of Impact Questions To demonstrate impact you must be able to eliminate other plausible explanations! 67

68 Internal Validity Goal of Design: Internal Validity Eliminates other plausible explanations 68

69 Internal Validity Internal Validity refers to the extent to which the design enables you to determine that the program, rather than other factors, caused the changes you have observed. 69

70 Threats to Internal Validity History: Maturation: Testing: Instrumentation: Event that took place during the period under study. Life cycle change (e.g., skills increase because people get older). Risk is that they learned how to do the test. Changes in data collection, Instrument, procedures or measures in pre/post or comparative designs. 70

71 Threats to Internal Validity Regression to the Mean: Selection: Attrition: Extreme scores on one measure of test are likely to be less extreme on the next measure of test. The group under study may be different in ways that effect the results Different rates of dropping from groups may effect results 71

72 Threats to Internal Validity These threats are possible. Does not mean they actually exist. You want to consider the probability that they do. 72

73 Importance of Design Design enables you to eliminate or reduce other possible explanations. 73

74 Key Design Elements that determine validity Can you control program implementation? set it up so you have measures before and after the program? find a comparison group? randomly assign people or things to receive the program? 74

75 Three types of designs Random assignment Before/ After Comparison group Experimental v v v Quasi experimental v v Non experimental?? 75

76 Types of Design Gender issues in Design: Men and women must be included (if applicable) in the design selected in the program and comparison groups in the before and after groups 76

77 Program Design Matrix Questions Sub-Questions Type of Question Type of Design Indicators Measures Data Sources Sample 77

78 Program Design Matrix Questions Sub-Questions Data Collection Data Analysis Comments 78