Workshop II Project Management

Size: px
Start display at page:

Download "Workshop II Project Management"

Transcription

1 Workshop II Project Management UNITAR-HIROSHIMA FELLOWSHIP FOR AFGHANISTAN 2007 Introduction to Project Management August 2007, Dehradun, India Presented by: Jobaid Kabir, Ph.D. Fellowship Program 2007 WS II 1 Part 7. Monitoring and Evaluation Fellowship Program 2007 WS II 2 1

2 Monitoring and Evaluation Monitoring and Evaluation basics Purpose of Monitoring and Evaluation Stakeholder Participation in Monitoring and Evaluation Planning and Managing an Evaluation Fellowship Program 2007 WS II 3 Steps of Planning and Managing an Evaluation 1. Planning Evaluations 2. Defining Evaluation Questions and Measurement Standards 3. Data Collection Process 4. Managing the Evaluation Process 5. Communicating and Using Evaluation Results 6. Evaluation Standards Fellowship Program 2007 WS II 4 2

3 In planning evaluation activities decide on WHY: purpose, users of evaluation findings and how WHAT: objectives and the questions to address HOW: data sources & collection methods WHO: will do evaluations, required expertise, need for external consultants, stakeholder involvement WHEN: timing of evaluations to help decision making RESOURCES: budget needs Fellowship Program 2007 WS II 5 Evaluation Analytical Process Evaluation Purpose Evaluation Objectives Overall Conclusions and lessons learned Overall Recommendations Analysis Synthesis Interpretation Assessment Questions Detailed Conclusions Detailed Recommendations Analysis Synthesis Interpretation Assessment Data Collection Methods Findings Specific Decisions Source: Danida, Fellowship Program 2007 WS II 6 3

4 Tools of Monitoring and Evaluation Performance indicators The logical framework approach Theory-based evaluation Formal surveys Rapid appraisal methods Participatory methods Public expenditure tracking surveys Cost-benefit and cost-effectiveness analysis Impact evaluation Fellowship Program 2007 WS II 7 Performance Indicators Measures: Inputs Processes Outputs Outcomes Enables: Tracking progress Demonstrate results Take corrective action Stakeholder participation improves their understanding and allows them to use indicators for decision-making Fellowship Program 2007 WS II 8 4

5 Use of Performance Indicators Setting targets and assessing progress toward achieving them Identifying problems through early warning system to take corrective action Tells whether an in-depth evaluation is needed Fellowship Program 2007 WS II 9 Advantages of Performance Indicators Effective means to measure progress towards objectives Facilitates benchmarking: With different organizational units With districts Over time Fellowship Program 2007 WS II 10 5

6 Disadvantages of Performance Indicators Poorly defined indicators are not good measures of success Tendency to define too many indicators, or those without accessible data sources, making system costly, impractical and likely to be underutilized Often a trade-off between picking the optimal or desired indicators and having to accept the indicators which can be measured using existing data Fellowship Program 2007 WS II 11 Cost for Performance Indicators Ranges from low to high Depends on: Number of indicators collected Frequency Quality of information sought Fellowship Program 2007 WS II 12 6

7 Skills Needs for Performance Indicators Well trained people for defining practical indicators MIS skills are required for implementing performance monitoring systems Fellowship Program 2007 WS II 13 Time Needed for Performance Indicators Depends on: Extent of participatory process Program complexity May take 6 12 months Fellowship Program 2007 WS II 14 7

8 Tools of Monitoring and Evaluation Performance indicators Logical framework approach Theory-based evaluation Formal surveys Rapid appraisal methods Participatory methods Public expenditure tracking surveys Cost-benefit and cost-effectiveness analysis Impact evaluation Fellowship Program 2007 WS II 15 The Logical Framework (LogFrame) Approach Helps clarify objectives of a project, program, or policy A vehicle for engaging partners in clarifying objectives and designing activities Serves as a useful tool to review progress and take corrective action Fellowship Program 2007 WS II 16 8

9 LogFrame Approach (Cont.) Helps identify expected causal links between inputs, processes, outputs, outcomes and impact Leads to identification of performance indicators at each stage in this chain, as well as risks which might impede the attainment of the objectives Fellowship Program 2007 WS II 17 Contents of a logical framework Summary Indicators Verification Assumptions Goal: Quantitative judgment of objectives achieved? (Estimated time) What sources of information exist, or can be provided cost-effectively? Necessary external factors for sustaining objectives? Purpose: Outputs: Produced by the project in order to achieve project purpose Quantitative measures of impacts and benefits. (Estimated time) Quantity of outputs and by when will they be produced? (Quantity, quality, time) What sources of information exist or can be provided cost-effectively? Does provision for collection need to be made under inputsoutputs? What sources of information? Necessary external factors if achievements of project purpose is to help reaching project goal? Factors outside the control of the project which, if not present, may limit progress from outputs to achievements of project purpose? Activities: Undertaken to accomplish outputs. External factors realized to obtain planned outputs on Indicators should be What are sources of schedule? included in all activities. information? Actions outside the control of the project necessary for Fellowship Program 2007 WS II 18 project inception 9

10 LogFrame Use Improve quality of project and program designs by requiring the specification of clear objectives, the use of performance indicators and assessment of risks Summarize design of complex activities Assist the preparation of detailed operational plans Provide objective basis for activity review, monitoring, and evaluation Fellowship Program 2007 WS II 19 Advantages of LogFrame Ensures that decision-makers ask fundamental questions and analyze assumptions and risks Engages stakeholders in the planning and monitoring process When used dynamically, it is an effective management tool to guide implementation, monitoring and evaluation Fellowship Program 2007 WS II 20 10

11 Disadvantages of LogFrame If managed rigidly, stifles creativity and innovation If not updated during implementation, it can be a static tool that does not reflect changing conditions Training and follow-up are often required Fellowship Program 2007 WS II 21 Cost for LogFrame Low to medium Depends on extent and depth of participatory process Fellowship Program 2007 WS II 22 11

12 Skills Needs for LogFrame Facilitation skill Advanced facilitation skills for participatory planning and management Fellowship Program 2007 WS II 23 Time needed for LogFrame Depends on scope and depth of participatory process Several days to several months Fellowship Program 2007 WS II 24 12

13 Tools of Monitoring and Evaluation Performance indicators The logical framework approach Theory-based evaluation Formal surveys Rapid appraisal methods Participatory methods Public expenditure tracking surveys Cost-benefit and cost-effectiveness analysis Impact evaluation Fellowship Program 2007 WS II 25 Theory-Based Evaluation Allows in-depth understanding of the workings of a program or activity the program theory or program logic. Need not assume simple linear cause-and-effect relationships Fellowship Program 2007 WS II 26 13

14 Theory-Based Evaluation (Cont.) By mapping causal factors and how they interact, it helps decide Which steps should be monitored How well they are in fact borne out Allows identifying critical success factors Fellowship Program 2007 WS II 27 Theory-Based Evaluation (Cont.) When data shows that these factors have not been achieved, the conclusion is that the program is less likely to be successful in achieving its objectives Fellowship Program 2007 WS II 28 14

15 Example of Theory-Based Evaluation Success of a program to improve literacy levels by increasing the number of teachers might depend on: Availability of classrooms and textbooks Likely reactions of parents, school administration and children Skills and morale of teachers Districts where extra teachers are to be located Reliability of funding Fellowship Program 2007 WS II 29 Theory-Based Evaluation Use Mapping design of complex activities Improving planning and management Fellowship Program 2007 WS II 30 15

16 Advantages of Theory- Based Evaluation Provides early feedback about what is or is not working and why Allows early correction of problems as they emerge Assists identification of unintended sideeffects of the program Helps prioritize issues to investigate in greater depth by using focused data collection or sophisticated M&E techniques Provides basis to assess the likely impacts of programs Fellowship Program 2007 WS II 31 Disadvantages of Theory- Based Evaluation Can become overly complex if the scale of activities is large or if an exhaustive list of factors and assumptions is assembled Stakeholders might disagree about which determining factors they judge important, which can be time-consuming to address Fellowship Program 2007 WS II 32 16

17 Cost for Theory-Based Evaluation Medium to high Depends on Depth of analysis Depth of data collection Fellowship Program 2007 WS II 33 Skills Needs for Theory- Based Evaluation Facilitation skill Analytical skill Fellowship Program 2007 WS II 34 17

18 Time needed for Theory- Based Evaluation Depends on Depth of the analysis Duration of the program or activity Depth of M&E work undertaken Fellowship Program 2007 WS II 35 Tools of Monitoring and Evaluation Performance indicators The logical framework approach Theory-based evaluation Formal surveys Rapid appraisal methods Participatory methods Public expenditure tracking surveys Cost-benefit and cost-effectiveness analysis Impact evaluation Fellowship Program 2007 WS II 36 18

19 Formal Surveys Can be used to collect standardized information from a carefully selected sample population Surveys often collect comparable information for a relatively large population in target groups Fellowship Program 2007 WS II 37 Formal Surveys Use Provides baseline data against which performance of the strategy, program or project can be compared Compares different groups at a given point in time Compares changes over time in the same group Fellowship Program 2007 WS II 38 19

20 Formal Surveys Use (Cont.) Compares actual conditions with the targets established in a program or project design Describes conditions in a particular community or group Provides a key input to a formal evaluation of the impact of a program or project Fellowship Program 2007 WS II 39 Advantages of Formal Surveys Findings from the sample of people interviewed can be applied to the wider target group or population Quantitative estimates can be made for the size and distribution of impacts Fellowship Program 2007 WS II 40 20

21 Disadvantages of Formal Surveys Results are often not available for a long period Data processing and analysis can be a major bottleneck for the large surveys Expensive and time-consuming Some kinds of information are difficult to obtain through formal interviews Fellowship Program 2007 WS II 41 Cost for Formal Surveys Ranges from roughly $30 60 per household in the United States Costs will be significantly higher if there is no master sampling frame for population to be surveyed Fellowship Program 2007 WS II 42 21

22 Skills Needs for Formal Surveys Sound technical and analytical skills for sample and questionnaire design, data analysis and processing Fellowship Program 2007 WS II 43 Time needed for Formal Surveys Depends on sample size May require 6 months to 2 years Fellowship Program 2007 WS II 44 22

23 Tools of Monitoring and Evaluation Performance indicators The logical framework approach Theory-based evaluation Formal surveys Rapid appraisal methods Participatory methods Public expenditure tracking surveys Cost-benefit and cost-effectiveness analysis Impact evaluation Fellowship Program 2007 WS II 45 Rapid Appraisal Methods A quick, low-cost way to gather the views and feedback of beneficiaries and other stakeholders, in order to respond to decision-makers needs for information Fellowship Program 2007 WS II 46 23

24 Rapid Appraisal Methods Use Provide rapid information for management decision-making for projects or programs Provides qualitative understanding of complex socioeconomic changes highly interactive social situations, or people s values and motivations Provide context and interpretation for quantitative data collected by more formal methods Fellowship Program 2007 WS II 47 Advantages of Rapid Appraisal Methods Low cost Can be conducted quickly Provides flexibility to explore new ideas Fellowship Program 2007 WS II 48 24

25 Disadvantages of Rapid Appraisal Methods Findings usually relate to specific communities or localities thus difficult to generalize Less valid, reliable and credible than formal surveys Fellowship Program 2007 WS II 49 Cost for Rapid Appraisal Methods Low to medium, depending on the scale of methods adopted Fellowship Program 2007 WS II 50 25

26 Skills Needs for Rapid Appraisal Methods Non-directive interviewing Group facilitation Field observation Statistical skills Fellowship Program 2007 WS II 51 Time needed for Rapid Appraisal Methods Four to six weeks Depends on size and location of the population interviewed and the number of sites observed Fellowship Program 2007 WS II 52 26

27 Rapid Appraisal Methods Key informant interview Focus group discussion Community group interview Direct observation Mini-survey Fellowship Program 2007 WS II 53 Rapid Appraisal Methods Key informant interview Series of open-ended questions posed to individuals selected for their knowledge and experience in a topic of interest Interviews are qualitative, indepth and semi-structured Fellowship Program 2007 WS II 54 27

28 Rapid Appraisal Methods Focus group discussion Facilitated discussion among 8 12 carefully selected participants with similar backgrounds Participants might be beneficiaries or program staff The facilitator uses a discussion guide Note-takers record comments and observations Fellowship Program 2007 WS II 55 Rapid Appraisal Methods Community group interview Series of questions and facilitated discussion in a meeting open to all community members The interviewer follows a carefully prepared questionnaire Fellowship Program 2007 WS II 56 28

29 Rapid Appraisal Methods Direct observation Use of a detailed observation form to record what is seen and heard at a program site Information may be about ongoing activities, processes, discussions, social interactions and observable results Fellowship Program 2007 WS II 57 Rapid Appraisal Methods Mini-survey A structured questionnaire with a limited number of close-ended questions that is administered to people Selection of respondents may be random or purposive (interviewing stakeholders at locations such as a clinic for a health care survey) Fellowship Program 2007 WS II 58 29

30 Tools of Monitoring and Evaluation Performance indicators The logical framework approach Theory-based evaluation Formal surveys Rapid appraisal methods Participatory methods Public expenditure tracking surveys Cost-benefit and cost-effectiveness analysis Impact evaluation Fellowship Program 2007 WS II 59 Participatory Methods Provide active involvement in decision-making for those with a stake in a project, program, or strategy and generate a sense of ownership in the M&E results and recommendations Fellowship Program 2007 WS II 60 30

31 Participatory Methods Use Learn about local conditions and local people s perspectives and priorities to design more responsive and sustainable interventions Identify problems and trouble-shooting problems during implementation Evaluate a project, program, or policy Provide knowledge and skills to empower public Fellowship Program 2007 WS II 61 Advantages of Participatory Methods Examines relevant issues by involving key players in the design process Establishes partnerships and local ownership of projects Enhances local learning, management capacity and skills Provides timely and reliable information for decision-making Fellowship Program 2007 WS II 62 31

32 Disadvantages of Participatory Methods Sometimes regarded as less objective Time-consuming if key stakeholders are involved in a meaningful way Potential for domination and misuse by some stakeholders to further their own interests Fellowship Program 2007 WS II 63 Cost for Participatory Methods Low to medium Costs vary greatly, depending on scope and depth of application and on how local resource contributions are valued Fellowship Program 2007 WS II 64 32

33 Skills Needs for Participatory Methods Facilitation skill Fellowship Program 2007 WS II 65 Time needed for Participatory Methods Varies greatly, depending on scope and depth of application Fellowship Program 2007 WS II 66 33

34 Commonly Used Participatory Tools Stakeholder analysis Participatory rural appraisal Beneficiary assessment Participatory monitoring and evaluation Fellowship Program 2007 WS II 67 Commonly Used Participatory Tools Stakeholder analysis The starting point of most participatory work and social assessments Used to develop an understanding of the power relationships, influence and interests of various people involved in an activity and to determine who should participate and when. Fellowship Program 2007 WS II 68 34

35 Commonly Used Participatory Tools Participatory rural appraisal A planning approach focused on sharing learning between local people, both urban and rural, and outsiders Enables development managers and local people to assess and plan appropriate interventions collaboratively often using visual techniques so that non-literate people can participate Fellowship Program 2007 WS II 69 Commonly Used Participatory Tools Beneficiary assessment Involves systematic consultation with project beneficiaries and other stakeholders to identify and design development initiatives, signal constraints to participation, and provide feedback to improve services and activities. Fellowship Program 2007 WS II 70 35

36 Commonly Used Participatory Tools Participatory M&E Involves stakeholders at different levels working together to identify problems, collect and analyze information and generate recommendations Fellowship Program 2007 WS II 71 Tools of Monitoring and Evaluation Performance indicators The logical framework approach Theory-based evaluation Formal surveys Rapid appraisal methods Participatory methods Public expenditure tracking surveys Cost-benefit and cost-effectiveness analysis Impact evaluation Fellowship Program 2007 WS II 72 36

37 Public Expenditure Tracking Surveys (PETS) Tracks flow of public funds and determine the extent to which resources actually reach the target groups Examines the manner, quantity, and timing of releases of resources to different levels of government, particularly to the units responsible for the delivery of social services Fellowship Program 2007 WS II 73 PETS (Cont.) Often implemented as part of larger service delivery and facility surveys which focus on the quality of service, characteristics of the facilities, their management, incentive structures, etc. Fellowship Program 2007 WS II 74 37

38 PETS Use Diagnosing problems in service delivery quantitatively Providing evidence on delays, leakage and corruption Fellowship Program 2007 WS II 75 Advantages of PETS Supports the pursuit of accountability when little financial information is available Improves management by pinpointing bureaucratic bottlenecks in the flow of funds for service delivery Fellowship Program 2007 WS II 76 38

39 Disadvantages of PETS Government agencies may be reluctant to open their accounting books Cost could be substantial Fellowship Program 2007 WS II 77 Cost for PETS Can be high until national capacities to conduct them have been established Fellowship Program 2007 WS II 78 39

40 Skills Needs for PETS Sound technical and analytical skills for sample and questionnaire design, data analysis and processing, and good understanding of sector to be assessed Fellowship Program 2007 WS II 79 Time needed for PETS Three to six months Fellowship Program 2007 WS II 80 40

41 Tools of Monitoring and Evaluation Performance indicators The logical framework approach Theory-based evaluation Formal surveys Rapid appraisal methods Participatory methods Public expenditure tracking surveys Cost-benefit and cost-effectiveness analysis Impact evaluation Fellowship Program 2007 WS II 81 Cost-Benefit & Cost-Effectiveness Analysis Cost-benefit and cost-effectiveness analysis are tools for assessing whether or not the costs of an activity can be justified by the outcomes and impacts Cost-benefit analysis measures both inputs and outputs in monetary terms Cost-effectiveness analysis estimates inputs in monetary terms and outcomes in non-monetary quantitative terms (such as improvements in student reading scores) Fellowship Program 2007 WS II 82 41

42 Cost-Benefit & Cost-Effectiveness Analysis Use Informing decision makers about the most efficient allocation of resources Identifying projects that offer the highest rate of return on investment Fellowship Program 2007 WS II 83 Advantages of Cost-Benefit & Cost-Effectiveness Analysis Good quality approach for estimating program and project efficiency Makes explicit economic assumptions that might otherwise remain implicit or overlooked at the design stage Useful for convincing policy-makers and funding agencies that the benefits justify the activity Fellowship Program 2007 WS II 84 42

43 Disadvantages of Cost-Benefit & Cost-Effectiveness Analysis Fairly technical, requiring adequate financial and human resources Required data may not be available and projected results may be highly dependent on assumptions made Results must be interpreted with care where benefits are difficult to quantify Fellowship Program 2007 WS II 85 Cost for Cost-Benefit & Cost- Effectiveness Analysis Varies greatly, depending on scope of analysis and availability of data Fellowship Program 2007 WS II 86 43

44 Skill Needs for Cost-Benefit & Cost-Effectiveness Analysis Require technical skill in economic analysis and availability of relevant economic and cost data Fellowship Program 2007 WS II 87 Time needed for Cost-Benefit & Cost-Effectiveness Analysis Varies greatly depending on scope of analysis and availability of data Fellowship Program 2007 WS II 88 44

45 Tools of Monitoring and Evaluation Performance indicators The logical framework approach Theory-based evaluation Formal surveys Rapid appraisal methods Participatory methods Public expenditure tracking surveys Cost-benefit and cost-effectiveness analysis Impact evaluation Fellowship Program 2007 WS II 89 Impact Evaluation Systematic identification of the effects positive or negative, intended or not on individual households, institutions and the environment caused by program or project Fellowship Program 2007 WS II 90 45

46 Impact Evaluation (Cont.) Helps understand the extent to which activities reach the poor and the magnitude of their effects on people s welfare Fellowship Program 2007 WS II 91 Impact Evaluation (Cont.) For large scale sample surveys project populations and control groups are compared before and after at several points during program intervention Fellowship Program 2007 WS II 92 46

47 Impact Evaluation (Cont.) For small-scale rapid assessment and participatory appraisals estimates of impact are obtained from combining group interviews, key informants, case studies and available secondary data Fellowship Program 2007 WS II 93 Impact Evaluation Use Measures outcomes and impacts of an activity and distinguishes these from the influence of other external factors Helps clarify whether costs for an activity are justified Fellowship Program 2007 WS II 94 47

48 Impact Evaluation Use Cont.) Tells decision makers whether to expand, modify or eliminate projects, programs or policies Draws lessons for improving design and management of future activities Fellowship Program 2007 WS II 95 Impact Evaluation Use Cont.) Compares effectiveness of alternative interventions Strengthens accountability for results Fellowship Program 2007 WS II 96 48

49 Advantages of Impact Evaluation (Cont.) Provides estimates of the magnitude of outcomes and impacts for different demographic groups, regions or over time Systematic analysis and rigor can give managers and policy-makers added confidence in decisionmaking Fellowship Program 2007 WS II 97 Advantages of Impact Evaluation Provides answers to central development questions: To what extent are we making a difference? What are the results on the ground? How can we do better? Fellowship Program 2007 WS II 98 49

50 Disadvantages of Impact Evaluation Some approaches are very expensive and time-consuming Reduced utility when decisionmakers need information quickly Difficult to identifying an appropriate alternative Fellowship Program 2007 WS II 99 Cost for Impact Evaluation Number of World Bank impact evaluations have ranged from $200,000-$900,000 depending on program size, complexity and data collection Rapid impact evaluations can be conducted for as little as $10,000- $20,000 Fellowship Program 2007 WS II

51 Training Needs for Impact Evaluation Strong technical skills in: Social science research design Management Analysis and reporting Quantitative and qualitative research skills Fellowship Program 2007 WS II 101 Time needed for Impact Evaluation Can take up to 2 years or more Rapid assessment evaluations can often be conducted in less than 6 months. Fellowship Program 2007 WS II

52 Examples Impact Evaluation 1. Randomized pre-test post-test evaluation 2. Quasi-experimental design with before and after comparisons of project and control populations 3. Ex-post comparison of project and non-equivalent control group 4. Rapid assessment ex-post impact evaluations Fellowship Program 2007 WS II Randomized pre-test post-test evaluation Design: Subjects (families, schools, communities etc) are randomly assigned to project and control groups Questionnaires or other data collection instruments are applied to groups before and after the project intervention Additional observations may be made during project implementation Fellowship Program 2007 WS II

53 1. Randomized pre-test post-test evaluation Example: Water supply and sanitation services where the demand exceeds supply Beneficiaries are selected by lottery Fellowship Program 2007 WS II Quasi-experimental design with before and after comparisons of project and control populations Design: Where randomization is not possible, a control group is selected which matches the characteristics of the project group as closely as possible Where projects are implemented in phases, participants selected for subsequent phases can be used as the control for the first phase project group Fellowship Program 2007 WS II

54 2. Quasi-experimental design with before and after comparisons of project and control populations Example: These models have been applied in World Bank low-cost housing programs in El Salvador, Zambia, Senegal and the Philippines Fellowship Program 2007 WS II Ex-post comparison of project and non-equivalent control group Design: Data collected on project beneficiaries and a non-equivalent control group is selected as Model Data collected only after project implementation Multivariate analysis is often used to statistically control for differences in the attributes of the two groups Fellowship Program 2007 WS II

55 3. Ex-post comparison of project and non-equivalent control group Example: Assessing the impacts of micro-credit programs in Bangladesh Villages where micro-credit programs were operating were compared with similar villages without these credit programs Fellowship Program 2007 WS II Rapid assessment ex-post impact evaluations Design: Some evaluations only study groups affected by the project while others include matched control groups Participatory methods can be used to allow groups to identify changes resulting from the project, who has benefited and who has not, and what were the project s strengths and weaknesses Fellowship Program 2007 WS II

56 4. Rapid assessment ex-post impact evaluations (Cont.) Design: Triangulation is used to compare the group information with the opinions of key informants and information available from secondary sources Case studies on individuals or groups may be produced to provide more indepth understanding of the processes of change Fellowship Program 2007 WS II Rapid assessment ex-post impact evaluations Example: Assessing community managed water supply projects in Indonesia Fellowship Program 2007 WS II