Program Evaluation. Mark Cipants

Transcription

1 Program Evaluation Mark Cipants

2 But isn t it just too hard? I can t measure the result of my program effort I don t know what information to collect If I can t calculate a return on investment, then its useless to evaluate Measurement is only effective in the production and financial areas. My CEO doesn t require evaluation so why should I? There are too many variables affective the behavioural change for me to isolate the impacts of training Evaluation will lead to criticism We don t need to justify our existence

3 As Psychologists... We are uniquely placed We have the expertise in evaluating behaviour change

4 Why evaluate? Determine if a program is accomplishing its objectives Identify strengths and weaknesses in HR/training processes To market future programs To determine if program was appropriate To calculate the return on investment To establish a database to help management make decisions

5 Process Diagram Intervention Performance Analysis Organisation mission, strategy, goals Work, organisational and competitive environment Source: ISPI 2001 Desired workforce performance GAP Actual state of workforce performance Evaluation Causal Analysis Lack of: Consequences, incentives and rewards Data, information, feedback Environmental support, resources, tools. Individual capacity Motives and expectations Skills and Knowledge Implementation and change management Appraisal systems Career development Coaching Culture change Compensation Documentation Environment Health, wellness Information systems Job aids Job, work design Leadership Organisational design Performance support Staffing Supervision Team building Training and education

6 Remember No need = No benefit

7 In the design phase, consider What is the intent? What benefits will flow? Who will benefit? What will be the value to the organisation? Consider tangible and otherwise

8 Benefits example Training in use of new computer processing system at a bank Beneficiary Benefit Value to organisation Frontline staff Management Customers

9 Integrated models of evaluation Model Levels Levels or key evaluation criteria Kirkpatrick 4 Reaction, learning, job behaviour, results Phillips 5 Birkenhoff 6 APSC/ANAO 6 Reaction and planned action, learning, application, business results, ROI Goal setting, program design, program implementation, immediate outcomes, usage outcomes, impact and worth Grouped into three phases: Line of sight (relevance and appropriateness), learning and performance, outcomes Source: Simon (2003)

10 Formative and Summative Evaluation The purpose of formative evaluation is to improve the delivery of the initiative - by considering how well the initiative operates against its planned methodology. The purpose of summative evaluation is to determine whether we should conduct the initiative at all by considering what impact the initiative has against its business objectives.

11 Formative Evaluation ASKS: How is the program operating and how can it be made better? What is the extent to which the program is reaching the appropriate target population? Is the service delivery consistent with the program design? Resources expended on need areas identified by program staff. Evaluator is often part of the program and works closely with program staff. Emphasis is on turning over useful information quickly to make improvements. Evaluation design is drawn up with staff and revised to meet their needs.

12 Formative Evaluation - typical questions Which aspects of the program worked well? Which aspects did not work so well? How can this program be improved? Would you recommend this to others? Was the program targetted at a suitable level? Was the content relevant? Was there a good mix of theory and practice? Was the material interesting? Were the exercises relevant?

13 Summative Evaluation The purpose of summative evaluation is to determine whether we should conduct the initiative at all by considering what impact the initiative has against its business objectives.

14 Summative Evaluation ASKS: Does the program achieve its objectives? How well has the program worked? Evaluation focuses on success variables considered important by decision makers. Information collection strategy will maximise external and internal validity and may be gathered over a longer period of time. The evaluation design is drawn up to meet the needs of key decision makers Evaluation purpose is to assess the impact of the program.

15 Outcomes Contribution to the business needs to be calculated Cost/benefit Return on Investment (ROI) Utility

16 Utility Analysis Alternative approach for calculating program s worth Formula: F = N(ExM) - C Where: F = financial utility N = number of participants E = effect or outcome of training program M = financial value of effect or outcome C = cost of program

17 Calculating Return on Investment Define cost of the program include development costs, program materials, instructor/facilitator costs, facilities, travel, lodging etc, participant salaries, admin/overhead costs, evaluation costs Define expected areas that should be impacted as a result of the initiative use benefits analysis Define measures that can tell you what has changed cost wise eg. Reduced errors Increased efficiency Increased sales Higher productivity Client retention Employee retention Devise means by which the training or program contribution can be determined. Return on investment = program benefits/program costs x100

18 ROI example Time saved in productivity gains from applying new project management skills per person = 5 days. 1 day saved = $227 net saving (based on daily, direct salary of someone on $80K p.y. not including on costs) 5 days saved per person = $1136 Savings from 40 participants who attended = 40 x $1136 = $ Program costs = $ Net ROI = $45 440/ x 100 = 354% Source: Simon (2003)

19 Activity You have been asked to complete a full evaluation to define the outcomes being achieved from a manager development program. Where would you start? What information would you collect? Why? What kinds of measures could you use to define appropriate outcomes?

20 ROI issues Need to be able to track both direct and indirect costs Program needs to be linked to some sort of tangible, costed outcome. Other benefits may not be readily apparent from this analysis eg. Morale, unless this can be directly linked to a tangible benefit. Intangible benefits should still be noted.

21 Major evaluation designs Single group post test only Single group pre and post test Time series design Simple control group design Experimental design

22 Other places to help gather data to isolate the impact Participant s estimate of impact (percent) Supervisor s estimate of impact (percent) Management s estimate of impact (percent) Use of experts Subordinate s report of other factors Calculating/estimating the impact of other factors Customer input

23 Validity issues in design Single group post test Single group pre and post test History Change Selection Mortality Test Effect Hawthorne X X X X X X X X X X X Time series X X X X X Simple control group Control group post test only Experimental design X X X X X X X Source: Smith (1992)

24 Source: Blanchard and Thacker Who will want this information? Reaction Learning Behaviour Outcome Trainer Yes Yes Yes No Other trainers Training manager Maybe Maybe Maybe No Yes Yes Yes Yes Trainees Yes Yes Yes Maybe Trainee s supervisor Upper management No maybe yes Yes No No maybe Yes

25 Considerations in evaluation Who is the likely audience for your evaluation? What will interest them? Which programs should be given a full evaluation?

26 Returns on investment summary of research Returns are almost always positive and can be quite high. Returns take many forms. Returns to are highest when the intervention is highly focused Has to be linked tightly to the specific needs of the business Source: Smith (2001)

27 Issues in evaluation Who should conduct the evaluation? Implementers? Managers? Consultants? Other HR/OD staff? Evidence v. proof of results