Indicators, Targets and Data Sources

Indicators, Targets and Data Sources

At the end of the session, participants will be able to: Construct Establish Describe Agree SMART Indicators Targets Good quality Data Sources on next steps 2

Results Management The art of result management is defining and achieving outcomes that are meaningful to both provider and client/consumer, are measured in a credible way, and are used in decision making- Susan Stout 3

Why Monitor, Report and Assess? Alerts managers to risks, problems and impediments to achieving results Forms basis for accountability within and across offices and for corporate reporting Encourages integration of lessons learnt into management decisions Provides inputs for validating technical and managerial policies and strategies. the benefits being? - improved programme delivery and achievement of ORs 4

Objectives Indicators Measures Targets Results Designing good Monitoring System-Key Questions what are we trying to achieve? What are we going to measure? How are you going to measure? What is the result that you want? What have you actually achieved? 5

Baseline Terms the situation prior to a development intervention against which progress can be assessed or comparisons made Indicator Target a quantitative and/or qualitative variable that allows the verification of changes produced by a development intervention relative to what was planned. a specific level of performance that an intervention is projected to accomplish in a given time period.

Baseline Target Achievement Baseline, Targets and Achievement Commitment Performance Current level of achievement 7

Baselines a clear picture of the pre-existing situation needed e.g. without knowing the baseline, how can one assess a 25% improvement in crop production baseline studies needed before targets can be set and before Approval can generally be given in some circumstances, an Inception Phase including some baseline data collection, may be appropriate 8

Indicators, Targets and Milestones Indicators are means; the proportion of girls achieving Grade 4 Targets are ends; increase by 15% in girls achieving Grade 4 by month 36

Types of Indicators Qualitative and Quantitative Combined both Qualitative and Quantitative Binary yes/no Direct and Indirect (Proxy) Product and Process

Indicators Indicators = How do we know?- Essential elements of the overall planning and monitoring system.

Indicators Indicators describe how the intended results will be measured - accountability Objectively verifiable, repeatable measures of a particular condition They force clarification of what is meant by the result.the fine print! Must be accompanied by baselines and targets

Logical Flow of Indicators Intervention Process and output Intermediate outcomes Final outcome (Impact) 13 30 September 2001

Performance Logic Inputs Process Outputs Outcomes Impact Funding, Planning & policies, Harmonization & efficiency Training & Capacity building, Procurement and supply Quality, Behavioural Interventions, & knowledge Service utilization and intervention coverage, Behaviour change Mortality, morbidity, nutrition So where are the results? 14

Indicators-Be Strategic! Be careful not to tie the hands of managers allow discretion and flexibility as performance-based incentives encourage innovation from the bottom up Golden rule: objectives of your program should guide choice of indicators Good indicators are outcomes closely correlated with impact, and outputs closely correlated with outcomes May consider process measures initially to establish critical systems (information, management, financial, etc.) 15

Indicators, Baseline, Target and Source of Data Outcome: By 2007, more girls in Belem Province enjoy a quality, basic education Output: 800 teachers in Belem Province can deliver the new curriculum effectively Indicator: Net enrolment ratio (M; F) Baseline: F:45% Target: F: 75% - Improvement in school test scores Indicator: # Teachers with new certification Baseline: 0 Target: 800 -Teacher proficiency reports -Improvement of school satisfaction ratings Source of Data -Annual school test scores report Source of Data: -School satisfaction surveys

Indicators and Targets: What s the Difference? Indicators A set of key measures that help you define and track progress toward your objective Targets Commitments, expressed in quantitative terms, of what you plan to achieve a specific statement of the amount of improvement to be achieved and the date by which it is to be achieved 17 Always set targets!

Targets: Guiding Principles Use targets to encourage improved performance and motivate people when there is a probability of success Targets should be realistic but challenging Too easy complacency Too difficult discouragement, demotivation Ensure targets at different levels are linked and define who is responsible for achievements 18

Tips for Targets Larger increase possible when starting from a low baseline Attainable Within a defined period Linked To performance 19

Performance Indicator Selection Criteria Validity - Does it measure the result? Reliability - Is it a consistent measure over time and, if supplied externally, will it continue to be available? Sensitivity - When a change occurs will it be sensitive to those changes? Simplicity - Will it be easy to collect and analyze the information? Utility - Will the information be useful for decision-making and learning? Affordable Do we have the resources to collect the information? Go beyond process to include outcome!

What is meant by:..enhanced policy....fully adopted by...capacity strengthened....fao s role is recognized....countries implementing...countries that have or are developing efficient and uniform services.. etc.? BUT in reality few result indicators are fully SMART They rarely stand-alone What does each clause and adjective and adverb mean in practice?

Typical Pitfalls Wordy (..and no change language) To promote equitable economic development and democratic governance in accordance with international norms by strengthening national capacities at all levels and empowering citizens and increasing their participation in decision-making processes Too ambitious Strengthened rule of law, equal access to justice and the promotion of rights Containing multiple results The state improves its delivery of services and its protection of rights with the involvement of civil society and in compliance with its international commitments

Typical Pitfalls Wishy-washy (ie. Support provided to improve..) Support to institutional capacity building for improved governance So general, they could mean anything To promote sustainable development and increase capacity at municipal level Overlapping with National goals/ MDGs (impacts) Substantially reduce the level of poverty and income inequality in accordance with the MDGs and PRSP Confusing means and ends Strengthen the protection of natural resources through the creation of an enabling environment that promotes sound resources management

Clarity Improves Reliability & Consistency Partners independently assessing the same change must come to the same conclusion No set of indicators is perfect; their limitations and weaknesses need to be acknowledged and allowed for Team members change; incoming staff must be able to clearly interpret indicators in the way intended by formulators Stakeholders may want clarification of, and justification for, the means of measuring, and the conclusions drawn

The need to clarify indicators Indicators need clarification eg. with a footnote or performance measurement matrix or explanatory note To take a well-known example: The MDG Indicator Handbook

MDG 5 - in FAO s remit By way of clarification: キ Definition キ Rationale キ Method of Computation キ Data Collection and Source

By way of clarification (cont): キ Periodicity of Measurement キ Gender Issues キ Disaggregation Issues キ Comments and Limitations Person or Agency responsible.

The need to clarify indicators To take another well-known example: The Paris Declaration Indicator 11

Tips for Indicators Have them; Align them to standards. Test them for validity and reliability. Be honest and unambiguous Discuss them and review them. Are they really key? Don t overestimate their importance! Benchmark them Do something with them. KPI should lead to change. Keep them simple but not too simple! How Many Indicators: No Perfect Answer:Too many can be counter-productive: difficult to communicate and manage, and can induce- indicator overload and fatigue Too few may focus attention and new resources on selected indicators at the expense of other important areas

Getting feedback Dialogue and data collection need prioritizing and resourcing. Allocate appropriate costs and time for getting it, usually from programme budget, rather than from other sources Agree means to open and maintain 2-ways channels, to survey users, to learn lessons and record success stories

Data Sources What evidence do we need? How do we get it? Available from existing sources? Is special data gathering required? Who will pay for data collection? How much data gathering is worthwhile? Who will collect and document the evidence? Where will it be located? 31

Examples of Data Sources records e.g. of secretariat; minutes, attendance lists, resolutions, budgets, accounts, etc. stakeholder feedback, results of focus groups documents, film, audio surveys and commissioned research reports annual reviews; harmonized partner reviews external evaluation reports local, national and global statistics and data 32

Humanize the Process Address staff concerns and ideas about performance management Address concerns about the uncertainty of the data Regardless of the topic, keep performance review processes fair and transparent Place performance review within a context of learning Enable a two way dialogue about performance Remember to highlight good news

Donors 34 For COWP Pilot Use Only

Q&A 35