ASSESSMENT OF EVALUATION THEORY: KIRKPATRICK MODEL IN OPPOSITION TO HAMBLIN MODEL

Transcription

1 ASSESSMENT OF EVALUATION THEORY: KIRKPATRICK MODEL IN OPPOSITION TO HAMBLIN MODEL Ms. Divya Sharma 1 1 Department of Management Science, Maharashtra Institute of Technology, Aurangabad (India) ABSTRACT Training includes any type of experience designed to facilitate learning which will aid performance in a present or future job. Evaluation at a learning level provides data on the degree of change to knowledge, skills or attitude from the program. This paper also talks about the principle of evaluation and its challenges. The paper also explores the Evaluation strategy model and global evaluation trend. This article reviews the approaches of Kirkpatrick model and Hamblin model into theoretical aspect. Meticulous attention is paid by comparing the variation between Kirkpatrick and Hamblin model. Keywords: Evaluation Strategy, Hamblin and Kirkpatrick model. I. INTRODUCTION Training and development programs are considered to be one of the important aspects of organizational development. There has been a growing need to find ways to determine the efficiency and effectiveness of such training programs both in terms of the organisation and the training institutions. Such evaluation would not only form the baseline for further progress, but would also help justify whether the investment in terms of time, money and energy has demonstrated its value to the organisation. Many Specialists from other countries, engaged in training and development, have tried to evolve different methods of assessing the effectiveness of training programs. Except in rare cases there seems to be no accepted method of establishing the cost benefit ratio because of evident limitations like the high variability and the long lead time in investment returns from it; turnover among the trained executives; newer and more urgent demands made on the trained executive which are beyond the scope of the programme; the motives of the sponsor, and the learner s attitude towards training which range from relaxation to serious learning. Kirkpatrick s 1 (1975) evaluation study extends and elaborates the evaluation process from mere reaction assessment to four stages, viz, reaction; learning; behavior and results of training. In spite of his approach to assessment at various stages, it precludes the trainee variables and organizational variable which could have a definite impact on the overall assessment of training and development. 267 P a g e

2 Hamblin 2 (1974) while linking up training objective with that of evaluation objectives, which may be directed towards a cost benefit approach, job related approach or the training centered approach, developed a conceptual scheme of evaluation which cuts through maze of evaluation stages enunciated by Kirkpatrick. Beside the objective setting and training phase Hamblin has also tried to draw guidelines for developing various strategies of management at various levels of evaluation. II. EVALUATION AND VALIDATION Evaluation means the assessment of value or worth. It would simply mean the act of judging whether or not the activity to be evaluated is worthwhile in terms of some criterion of value, in the light of the information available. Evaluation has traditionally been taken to include not only the assessment of value, but also the collection and analysis of the information on the basis of which the assessment is to be made. This broader definition incorporates the objective of evaluation which includes investigation before and after training as well as during the training because one cannot assess training efforts unless something about the before and after training situation is known. Validation is defined in terms of Internal and External validation. Internal Validation deals with a series of tests and assessments designed to ascertain whether a training program has achieved the specified objectives. External validation is a series of test designed to ascertain whether the objectives of an internally valid program are realistically based on an accurate initial identification of training needs in relation to the criteria of effectiveness adopted by the organization. The term evaluation involves measurement in social and financial terms which may not be tangible for external training programs. Validation is limited to lay down objectives excluding the anticipated and unanticipated effects, both desirable and undesirable. For the purpose of research the broader definition of evaluation partly based on Hamblin s definition appears appropriate. Evaluation could thus be defined as an attempt to obtain information or feedback on the effects of a training program, and to assess the value of this training program. 2.1 Principles of Evaluation To be effective, evaluation of training must be consistent with the purposes, objectives and goals of the training activity based on the following guidelines: Trainer must be clear about the goals and purpose of evaluation Essential involvement of the training staff, the trainer and the sponsoring organisation Evaluation must be continuous and specific Evaluation must provide the means and focus for trainers to be able to appraise themselves Evaluation must be based on objective methods and standards Effective communication and coordination are essential Realistic target dates must be set for each phase of the evaluation 268 P a g e

3 2.2 Problems in Evaluation 1. Lack of Goal Congruence The main challenge is difference in perception about the objectives of the training, they basically lack in clarity of specific training objectives. 2. Trainers, Organization and Trainees approach to evaluation Trainers use his past experience and some kind of intuitive feeling about the course rather than employing a systematic method of feedback to improve subsequent programs. Management does not establish clear objectives for training but remain vague and ambivalent. The trainee as evaluator may perceive himself to be in a very superior role of being appraiser. 3. Problems arising from the nature and variability of management tasks Variety of factors influencing the behavior of managers, they may include individual factors, personality, motivation, knowledge, skills and attitudes, organizational climate, cooperation of subordinates and colleagues and the type and nature of industry. 4. Problems arising from the Quantitative approach to Evaluation Rigid mathematical systems without specific application to reality while the simply refuses to consider the problems of quantification. 2.3 Evaluation Strategy Model Evaluation of training is an important activity because a training program may have to be continuously cut, lengthened, changed, resequenced, rescheduled, eliminated or even retained as it is, on the basis of the evaluation findings. Although at first glance evaluation findings determine how well a training program is achieving its goals and objectives, ultimately a good evaluation system has to be source of information upon which a variety of decisions can be made and purposes served. Evaluation can be viewed as a method of measuring change in knowledge, skills, attitudes, job performance, costs and the quality of training facilities. The process of evaluation evolve certain criteria Setting training objectives Selecting trainees for the training program Linking the training inputs with the goals and objective of the training Arriving at the suitable pedagogical tools and techniques Measuring the impact of training programs Determining the methods of review and feedback Organization can explore and correlate various biographical and organizational factors related to i.e. training to comprehend the process of learning. Transfer of training to the job so as to arrive at guidelines for optimizing the training effect. This will help identify the organizational variables which help hinder the learning process so that by overcoming these hindering factors the organisation can make constructive use of the training effort. 269 P a g e

4 2.4 Global Evaluation Trend Measurement and evaluation have been changing and evolving in private and public sector, as well as nongovernmental organizations. This change is occurring across organizations and cultures worldwide. The following are a few of the most obvious trends. Organizations are increasing their investments in measurement and evaluation, with best practice groups spending 3 to 5 percent of the HRD budget on measurement and evaluation. Organisations are moving up the value chain, away from measuring reaction and learning to measuring application, impact, and ROI Needs of the clients and sponsors of training and development projects, programs, initiatives, and solutions are the primary drivers for the increasing focus on measurements and evaluation. Evaluation is an integral part of the design, development, delivery, and implementation of programs. A shift from a reactive approach to a proactive approach is occurring as organization address evaluation early in the cycle. Measurement and evaluation processes are systematic and methodical, often a seamless part of the delivery process. Technology is significantly enhancing the measurement and evaluation process, enabling the data collection, processing, analysis, and integration of large amount of data. Evaluation planning is becoming a critical part of the measurement and evaluation cycle. The implementation of comprehensive measurement and evaluation processes usually leads to emphasis on initial need analyses. Organization with comprehensive measurement and evaluation systems in place has seen increases in their program budgets. Organization without comprehensive measurement and evaluation systems see the reduction or elimination of their program budgets. The use of ROI is emerging as an essential part of many measurement and evaluation systems. It is a fast growing metric 70 to 80perecent of organization have it on their wish lists. Many successful examples of comprehensive measurement and evaluation applications are available in all types of the organization and cultures. 2.5 Kirkpatrick model Kirkpatrick developed his four step model in 1959 and provided a simple and pragmatic model for helping practitioners think about training programs. Kirkpatrick (1977) divided the evaluation model into four parts: reaction; learning; behavior and results. Reaction would evaluate how participants feel about the programme they attended. The learning would evaluate the extent to which the trainees learned the information and skills, the behavior would evaluate the extent to which their job behavior had changed as a result of attending the training. The results would evaluate the extent to which the results have been affected by the training 270 P a g e

5 programme. According to a survey by the American Society for training and development (ASTD), the Kirkpatrick four level evaluation approaches is still the most commonly used evaluation framework among Benchmarking Forum Companies (Bassi & Cheney, 1997). It has been criticized for implying a hierarchy of value related to the different levels, with organizational performance measures being seen as more important that reactions. More fundamentally, there have been criticisms of the assumptions that the levels are each associated with the previous and next levels. This implied casual relationship has not always been established by research. The main strength of the Kirkpatrick evaluation approach is the focus on behavioural outcomes of the learners involved in the training (Mann & Robertson, 1996).The strengths of this model lie in its simplicity and realistic way of helping practitioners think about training programs (Alliger and Janak, 1989). The best known and most widely used framework for classifying evaluation is the Kirkpatrick model. The model consists of four stages, originally described more recently by Kirkpatrick (1996) as levels. The four levels are: Level Dimensions Descriptions What participants thought of the program, measured by the use 1 Reaction of reaction questionnaire Changes in Knowledge,skill and attitude,, assessed by use of 2 Learning performance tests Changes in job behavior and to identify learning are applied. 3 Behavior Assessment method observation and productivity data. Contribution of the training program, methods includes ROI, 4 Results measuring costs. Source: from Kirkpatrick, P a g e

6 Let us discuss the four level of Kirkpatrick Level 1 Reaction: Reaction evaluation is to enhance the quality of training programmes, which in turn leads to improved performance by measuring the participant s reactions to training programme. This should be measured immediately after the programme. Level one evaluation should not just include reactions towards the overall programme (e.g. did you like the programme.); it should also include measurement of participants reactions or altitudes towards specific components of the programme such as, the topics, contents, methodology, instructor etc. Level 2 Learning: Evaluation at this level wants to differentiate between what they already knew prior to training and what they actually learned during the training programme (Jeng & Hsu, nd.). Learning outcome can include changes in knowledge, skills or attitudes. The Evaluation should focus on measuring what was covered in the training events i.e. learning objectives. Level 3 Behavior: Behavior evaluation is the extent to which the trainees applied the learning and changed their behavior, and this can be immediately and several months after the training, depending on the situation. This level evaluation wants to measure the transfer that has occurred in the learner s job behavior/ job performance due to the training programme. Level 4 Results: The intention at this level is to assess the coat vs. benefits of training programme, i.e. organizational impact in terms of reduced costs, improved quality of work, higher productivity, reduction in turnover, improved human relation, increased sales, fewer grievances, lower absenteeism. Higher work morale, fewer accidents, greater job satisfaction etc. Collecting, organizing and analyzing level four information can be difficult, time consuming and more costly than the other three levels, but the results are often quite worthwhile when viewed in the full context of its value to the organisation. Kirkpatrick (1959) originally discussed reactions in terms of how well participants liked a particular program. In practice, measures at this level have evolved and are most commonly directed at assessing trainees affective responses to the quality (e.g. satisfaction with the instructor) or the relevance (e.g. work-related utility) of training. Learning measures, level two, are quantifiable indicators of the learning that has taken place during the course of the training. Level three behavior outcomes address either the extent to which knowledge and skills gained in training are applied on the job or result in exceptional job-related performance. Finally, level four outcomes are intended to provide some measure of the impact that training has had on broader organizational goals and objectives. In recent practice, the typical focus of these measures has been on organizational level financial measures. 272 P a g e

7 It has provided straightforward system or language for talking about training outcomes and the kinds of information that can be provided to assess the extent to which training programs have achieved certain objectives. Kirkpatrick insisted that information about level four outcomes is perhaps the most valuable or descriptive information about training that can be obtained Critics on Kirkpatrick Model Expert Dan McCarthy says that there is low correlation among the four levels (reaction, learning, behavior, and results). This takes nothing away from the contribution that the Kirkpatrick model has made to the field over the past 50 years. McCarthy suggests, we only compare what people knew before training to what they know and do after training, we still will not know what to do to achieve better results. Bernthal (1995) argues that the model mixes evaluation and effectiveness and that these do not form a continuum. Many evaluation studies that have evaluated training on two or more of Kirkpatrick s levels have reported different effects of training for different levels (Alliger and Janak, 1989). Holton (1996) argues that the levels form taxonomy of outcomes rather than a model. Kraiger and Jung (in Quinones,1997) agree with his view and argue that at the same time Kirkpatrick provides a model for thinking about how to evaluate, it does little to inform what to evaluate and how to link the results to strategy. Other complaints are that the model is too simple and fails to take account of the various intervening affecting learning transfer. Percentage that the Kirkpatrick Model is used: Bassi, et al (1996) reported that 96% of companies surveyed used some form of the Kirkpatrick framework to evaluate training and development programs. Twitchell, Holton, and Trott (2000) performed a meta-analysis of studies performed in the last 40 years. Their research indicates the following ranges for the use of Kirkpatrick's four levels: Level 1: % Level 2: 71-90% Level 3: 43-83% Level 4: 21-49% There is no doubt that Kirkpatrick s model has made valuable contributions to training evaluation thinking and practice. It has helped focus training evaluation practice on outcomes (Newstrom, 1995), fostered the recognition that single outcome measures cannot adequately reflect the complexity of organizational training programs, and underscored the importance of examining multiple measures of training effectiveness. The model promoted awareness of the importance of thinking about and assessing training in business terms (Wang, 2003). 273 P a g e

8 III. THE FIVE LEVEL APPROACH : HAMBLIN (1974) Hamblin was one of the first to modify Kirkpatrick s model. The first three levels in his model correspond closely to Kirkpatrick s model. However, the final level is split into two organization and ultimate value. Unlike Kirkpatrick, Hamblin suggests that the five levels of his model form a hierarchy. The five level models are therefore: Level Dimensions Descriptions 1 Reaction What participants thought of the program, measured by the use of reaction questionnaire 2 Learning Changes in Knowledge,skill and attitude,, assessed by use of performance tests 3 Job Behavior Changes in job behavior and to identify learning are applied. Assessment method observation and productivity data. 4 Organization Effect on organization, from participants job to performance changes 5 Ultimate value The financial effects, both on the organizational and the economy. Hamblin framework is intended to supplement other evaluation classification. This framework was first presented in an earlier article (Hamblin, 1968), but it has been modified and improved a lot. It has proved useful on a number of training courses, on which it helped trainees to see the connections between widely differing approaches to training evaluation, and to plan evaluation strategies for their own firms. The framework is presented in diagrammatic form as below: 274 P a g e

9 3.1 Hamblin cycle of evaluation ( O represents Objective and E represents Effects) In Hamblin model we are assuming that there is a cause and effect chain linking the five levels of training effects. This can be set out as follows: TRAINING which leads to REACTIONS which lead to LEARNING which leads to CHANGES IN JOB BEHAVIOUR which lead to CHANGES IN THE ORGANIZATION which lead to CHANGES IN THE ACHIEVEMENT OF ULTIMATE GOALS. This chain may snap at any of its links. Though first three level is already discussed in Kirkpatrick model, let us discus the same through Hamblin model Level 1 Reaction: The effect after training is that the trainees will react to the training and their reactions will be highly complex and shifting over time. According to his model 2 variable will play an important role i.e. reactions effects and reactions objective. We must be selective and must have criteria for selection. This means in effect that we must determine the reactions in which we are interested in investigating that how trainees will react. Level 2 Learning: Learning depends on reaction; people who react inappropriately will fail to learn inappropriately. Trainees existing state of learning (knowledge, skill and attitude) must be compatible and should be receptive to the training. Trainees must have basic aptitude. Level 3 Job Behavior: In other circumstances, it may be possible and worthwhile to go beyond the terminal behavior of the trainees, and to find out whether they have applied their learning on the job. The ultimate objective of the training program is job behavior and the training is regarded as successful if the desired behavior changes are achieved. Level 4 Organization: Hepworth (1972) suggests that all organizations have four primary objectives, which are in order of primacy. 1. Survival of the organization 2. Creation of surplus money 3. Welfare of interested parties 4. Social / Political welfare Therefore organization at level 5 i.e. Ultimate values must decide to objective of the training to contribute towards the organization and weighting to give to the different objectives. Level 5: Ultimate Value Training should be defined in terms of the trainee s own personal goals( improved financial reward, job opportunity and self esteem ) rather than those of the organization. Ultimate value should not be seen purely in terms of profit or cost effectiveness. Uses of the Hamblin model: 275 P a g e

10 The model can be seen in two ways. 1. It can be seen as a cycle of events occurring in an organization: training objectives lead to training, which leads to training effects, which lead to more training objectives. 2. It can be seen as a cycle of evaluation activities the trainer/ evaluator is engaged, in which every activity apart from the training activity itself is a stage in the process of training. The purpose of the model is to help the manager or trainer to decide which activity or activities he is already involved in, and to plan the route which he should follow from there. IV. CONCLUSION The purpose of evaluation is to create a feedback loop or a self correcting training system (Rackham, Honey, and Colbert, 1971). A well controlled training program is one which weakness and failures are identified and corrected by means of negative feedback, and strengths and success are identified and amplified by means; as there are many Evaluation model. It is observed that basically Kirkpatrick framework is considered as core. If we see the Hamblin model first three levels is derived from Kirkpatrick framework or expanded model which encapsulates certain elements both before assessing reactions and after an examination of organizational results. Though there are many critics about the Kirkpatrick Model, many of the organization uses the model for their training evaluation. REFERENCES 1. Blanchard PN, Thacker JW,Way SA(2000), Training evaluation : perpestives and evidence from Canada, International Journal of Training and Development, Vol. 4(4),pp J. Philips, Handbook of training evaluation and measurement methods (Butterworth- Heinemann, Oxford, 1991) 3. R. A. Noe, Employees Training and Development (Irwin, Mc Graw Hill, 2008). 4. L. M. Prasad, Human Resource Management (New Delhi, Sultan Chand & Sons, 2005) 5. Kirkpatrick DL(1983), Four steps to measuring training effectiveness, Personnel Administrator, Vol. 28(11),pp Kirkpatrick DL(1996), Great ideas revisited : revisiting Kirkpatrick four level model, Training and Development, Vol.50(1), January,pp Philips, J. J. (1996c). How much is the training worth? Training & Development, 50(4), Ulrich, D. (1997). Measuring human resources: An overview of practice and a prescription for results. Human Resource management, 36(3), A Hamblin. Evaluation and control of training. McGraw-Hill, London, P a g e