Why & how performance management & evaluation. OECD Trento Centre 10 th 13 th November

Transcription

1 Why & how performance management & evaluation OECD Trento Centre 10 th 13 th November

2 This presentation covers Why performance manage & evaluate The benefits, the risks The choices of approach How to plan, deliver and evaluate performance Some hints lessons from the field My evidence 25+ years practice

3 How does it seem to you? Doing the right things? Doing enough of them? Do them better? Benefits, costs, impacts? Will impact last?

4 Why do it? If we don t know where we are going any way will get us there If we don t know where we are going, how will we know when we get there? Getting there together

5 Benefits & risks Benefits. Evidence of activity Learn from experience Strengthen joint work Problem solving Improve impacts Move resources to high performing activities Risks. Measurement is misleading Becomes the purpose Slow, saps energy Leaders are inexperienced in use of evidence Look to experts

6 Preconditions of success Clarity about shared outcome Responsive governance Systemic model for change Fit for purpose information flows

7 Importance of cause & effect Risk of changing symptoms, rather than underlying conditions How & where do you intervene, in order to achieve what? Trends and sustainable impacts

8 Important concepts Outcomes Outputs Baselines Trend & forecast Indicator Targets (# & %) Impacts

9 What do you think, so far? How does this compare with your experience? New ideas? Things you might change? Things over looked? What s real?

10 Remember the important concepts? Outcomes Outputs Baselines Trend & forecast Indicator Targets (# & %) Impacts

11 Your theory of change What do you all agree will create change? What s your evidence for this? What s the depth of agreement & understanding? Triangulated how?

12 Levels & contexts Levels of Population Complexity Maturity / experience Contexts History Life cycle Persistence of problems Etc etc Fitness for purpose

13 3 stage process At centre of programme design Informs policy cycle Performance and impact in mind from outset Critical friend EU models

14 Generic indicators Eg SME sectoral support Limit # of indicators (max 8?) Relate actions to causes & effects Logic chain in actions, impacts, indicators

15 A system of indicators How indicators link with activity Assumptions Virtuous circle of improvement Add in risk Informs choice & modifies behaviour

16 Planning it EU style Management plan for how performance & evaluation relate Multi phase / multi annual Multi project focus within a programme Hierarchy of systems

17 Things Things to avoid if to avoidyou can Managing resources vs performance Indicators with no baseline Perverse effects Weak feedback & feed through Slow responsiveness

18 Planning & structuring Concept / issue mapping Stakeholder consultation Evaluability assessment Logic models Formative / developmental

19 Obtaining data Surveys (social & beneficiary) Individual stakeholder interviews Priority evaluation Focus groups Critical incident analysis Case studies Data admin & 2 nd ry Observation Group consultation

20 Qualitative or quantitative? Informs subtle processes eg behaviour change Interest in contexts Human judgements Bottom up understanding Explaining causal patterns Impacts for different groups Innovation & explanations for differential impacts

21 Qualitative or quantitative? Aggregate judgements Explanatory & predictive modelling Overview extent & scale Comparisons across settings / client groups Particular interventions Trend analysis

22 What do you think, so far? How does this compare with your experience? New ideas? Things you might change? Things over looked? What s real?

23 Analysing information Input:output Econometric Regression analysis Experimental Delphi expert feedback SWOT

24 Cost benefit analysis Benchmarking Cost effectiveness Economic impact Equalities impact Environmental impact Strategic Multi criteria Expert panels Tools to use

25 From experience. Outcomes Real focus on causes Deep agreement on activities Small no. indicators Rates of change & absolute numbers Plausibility

26 .and also from experience Forensic interest in delivery chains How does the customer experience the service? Clout in governance Is the story, change?

27 Baselines Where are we heading? What scenarios and influencing factors are there? How will our targets relate to the future? So what? The Matter of Baselines H M OK? L O O Point to Point Turning the Curve History Forecast Baselines have two parts: history and forecast

28 Discussion examples Based on 2 thumbnail cases studies How does this compare with your experience? New ideas? Things you might change? Things over looked? What s real?

29 E ffect E ffo rt Programme performance Different types of performance How much? How well? How many benefit? What proportion? So what? P rogram m e P erfo rm ance M easures Q u antity H o w m u ch d id w e d o? Is anyone better off? Q u ality H o w w ell d id w e d o it? # %

30 Effect Effort Quantity How much did we do? Quality How well did we do it? Least Important 2 nd Most Important Is anyone better off? 3 rd Most Important Most Important

31 How much did we do? # Clients/customers served # Activities (by type of activity) # # # # Point in Time vs. Point to Point Improvement Is anyone better off? How well did we do it? % Common measures e.g. client staff ratio, workload ratio, staff turnover rate, staff morale, % staff fully trained, % clients seen in their own language, worker safety, unit cost % Activity-specific measures e.g. % timely, % clients completing activity, % correct and complete, % meeting standard % Skills / Knowledge (e.g. parenting skills) % Attitude (e.g. toward drugs) % Behavior (e.g.school attendance) % Circumstance (e.g. working, in stable housing)

32 a drinking example Drug/Alcohol Treatment Programme Quantity How much did we do? Number of persons treated Number of clients off alcohol & drugs - at discharge - 12 months after discharge Is anyone better off? Quality How well did we do it? Percent of staff with training/ qualification Percent of clients off alcohol & drugs - at discharge - 12 months after exit Different types of indicators tell different stories They inter-relate to provide the whole picture

33 Thanks for listening Thoughts? Feelings? Decisions to act? How can performance management systems help change behaviour for the better/ best?

34 Resources on line gional_policy/sources/ docgener/evaluation/e valsed/index_en.htm gional_policy/sources/ docoffic/working/sf20 00_en.htm

35 Andrew Harrison +44(0) ning-studio.org inking studio_style/ Happy to help