Learning Objectives. Explain the role of evaluation in the workplace Discuss how an evaluation process can be carried out in the workplace.

Transcription

1 1

2 Lecture Outline (1) The Importance of Evaluation (2) Arguments for and against measurement (3) Aims of evaluation (4) Best Practice Models of Evaluation 4a) Hierarchical Models of Evaluation 4b) Steps in carrying out an Evaluation 4c) Types of Questions for gathering subjective responses 4d) Evaluating Costs 2

3 Learning Objectives Explain the role of evaluation in the workplace Discuss how an evaluation process can be carried out in the workplace. 3

4 The Importance of Evaluation What is the importance of evalution of training programme? 4

5 The Importance of Evaluation Evaluation tells us whether the HRD effort was successful or not. It determines the worth of a training program. This is not only related to the amount of money spent on training. Due to its cumbersome nature, evaluation is often carried out reluctantly or superficially in the workplace. 5

6 The Importance of Evaluation As organisations increasingly spend more on training and with the search for sustainable competitive advantage, the focus on evaluation has increased. Some other reasons why HRD needs to evaluated is efforts are: 6

7 The Importance of Evaluation (cont..) Increasing interest in demonstrating the impact of HRD To disprove the perception that little is done in evaluation when compared to the investment in training. A greater emphasis on outcomes and evaluation will show the outcomes. 7

8 Arguments for and against measurement Evaluation is ultimately about measurement. The following table shows some of the main arguments for and against measurement of HRD efforts. Arguments For Helps credibility Provides objective data Provides a reflection of reality Enables marketability May help to maintain quality Demonstrates success Emphasizes financial measures Links with setting goals. Arguments Against Can be disruptive and demoralizing Outcomes are subjective Difficult to prove cause and effect Motivation maybe political May diminish trust May breed complacency Not necessarily human performance What gets measured gets done 8

9 Aims of evaluation Organizations conduct needs analyses for different purposes. Some of these purposes are: Demonstrate contractual compliance Legitimize the approach to development Demonstrate worthwhile benefit in relation to cost and provide feedback on investment Drive change 9

10 Best Practice Models of Evaluation Traditionally, evaluation was based on hierarchical models which have now been developed to include greater flexibility. In this study guide we discuss two models of evaluation: the hierarchical models and the Kirkpatrick model. 10

11 Hierarchical Models of Evaluation Hierarchical models of evaluation identify the levels at which evaluation is made. The following model by Reid and Barrington, 1997 highlights 5 levels. Level Level 1 Level 2 Level 3 Level 4 Level 5 Evaluation Reaction Learning Job behavior Departmental gain Ultimate level organization 11

12 Hierarchical Models of Evaluation (cont..) We tend to measure the reaction levels in HRD efforts by handing out evaluation forms at the end for training programs. The learning level is also considered by assessing the learning outcomes through a test or other source. 12

13 Hierarchical Models of Evaluation (cont..) The last three levels are not widely evaluated in the workplace. In addition to evaluations reaction and learning, it is important for organisations to evaluate job behaviour (transfer of learning) and then to go broader to evaluate the impact on the department and the organisation. 13

14 Hierarchical Models of Evaluation (cont..) The Kirkpatrick model, widely used in evaluation condenses the above 5 levels to 4. The following table gives details on the levels and how they can be measured. Level What is measured How it is measured Level 4: Results Organizational benefits resulting from training Financial reports Quality inspections Level 3: Behavior The extend of behavior change in the workplace resulting from training Completed selfassessment questionnaire On-the-job observation Reports from customers, peers and participant s manager 14

15 Hierarchical Models of Evaluation (cont..) Level What is measured How it is measured Level 2: Learning Level 1: Reaction The extent to which participants improve knowledge and skills, and change attitudes, as a result of training Participants reaction to program Pre and post-test scores On-the-job assessments Supervisor reports Completed participant feedback questionnaire Informal comments from participants Focus groups with participants 15

16 Steps in carrying out an Evaluation The following steps offer a guide as to how a summative evaluation or an evaluation at the end of the HRD initiative can be carried out. (Kemp et al, 1998: ) 1. Specify program objectives. 2. Determine the evaluation design for each objective a. Pretest post test with one group. (people participating in training are given a test prior to the program and after the program. Results of the two tests are compared to determine additional learning). 16

17 Steps in carrying out an Evaluation b. One-group descriptive c. Experimental control group (the group who underwent training are measured as well as a control group that didn t go through the training. Results of the tests are compared to determine if the experimental group achieved learning) d. Analysis of costs, resource, implementation 17

18 Steps in carrying out an Evaluation (cont..) 3. Develop data-collection instruments and procedures for each objective. a. Surveys b. Interviews c. Observations d. Achievement tests 4. Conducting the evaluation a. Schedule data collection b. Collecting data 18

19 Steps in carrying out an Evaluation (cont..) 5. Analyze results from each instrument 6. Interpret results 7. Disseminate results and conclusions a. Evaluation report b. Group meetings c. Individual discussions 19

20 Types of Questions for gathering subjective responses The following types of questions can be used for gathering subjective responses such as how satisfied trainees are with the program in question. 1) Check lists Example: Check each word that tells you how you feel about the group work used in this program. Dull Useful Exciting Difficult Important Unpleasant Boring Practical Interesting 20

21 Types of Questions for gathering 2) Rating Scale subjective responses Rating scale type of questions asks the trainees to rate either according to a verbal or numerical scale. An example of a verbal scale is: How useful was the content covered in the program. Useless somewhat useful very useful An example of a numerical scale is: On a scale of 1 to 5, with 1 being extremely dissatisfied and 5 being extremely satisfied, how would you rate your satisfaction with the program?

22 Types of Questions for gathering 3) Ranking subjective responses In a raking type of question, trainees are asked to rank items in answer to a question. Example Please rank the following content, covered in the program on leadership, in order of importance to your job. (Start with number 1 as the topic with the most value) Planning Team Building Decision Making Performance Management Stress Management Budgeting 22

23 Types of Questions for gathering subjective responses 4) Open ended questions As the name implies, open ended questions would seek open, detailed answers from trainees. Example: What is your general impression about this program. You can comment on the objectives, content, delivery and the overall arrangements of this program. 23

24 Evaluating costs One area that top management who invests in training and HRD is most interested in, is report on the costs of implementing the program. Two categories of costs can be considered in evaluating the total costs of the program. 1) Developmental costs 2) Operational costs 24

25 Evaluating costs 1) Developmental costs These are the costs that are incurred in developing the course. This is also called start-up costs and can include: Planning time: can be calculated based on the amount of time planning for the program and costing this time based on the salary of the people who were involved in this planning. 25

26 Evaluating costs 1) Developmental costs Supplies and materials for preparing material for the program. Equipment purchased or rented for the program. (e.g.: laptops, projectors) Overhead: utilities, furniture etc. 26

27 Evaluating costs 2) Operational costs These are costs incurred when the program is being implemented and includes recurring operational costs. Operational costs could include the following: Consultancy fees, accommodation and food expenses for external lecturers or trainers 27

28 Evaluating costs 2) Operational costs Learner or trainee costs: costs of their salary, based on time spent in program. Rental charge for conducting the sessions of the program Overhead: utilities, facilities. Refreshment costs for trainees and lecturers during the session 28

29 Q & A 29